Jump to content


Photo
- - - - -

Replacement for convert.php script


  • Please log in to reply
40 replies to this topic

#1 Tigra

Tigra

    Member

  • Members
  • PipPip
  • 17 posts

Posted 01 November 2007 - 10:12 AM

Looking through code of ClipShare, i've figured out that movie converson routine is somewhat buggy and not optimal. So i've wrote script as a replacement of default convert.php/upload.php scripts.

Benefits:

Written in Python, cross-platform (tested and seems to work fine on FreeBSD, Linux, Windows (XP, 2000, 2003))

Designed to run as background daemon process, it looks time to time into special 'request' database table, picks least recent record and performs conversion. Such approach are a lot better than executing 'convert.php' in background as currently implemented in ClipShare, because:
a) workload on video conversion server is constant through time (only one job at time is running opposing to as many as requested in current CS version)
B) doesn't require GD library to generate movie thumbnails (it uses mplayer's ability to extract JPEGs from movie stream and downsample it to specified size)
c) Have small memory footprint and able to run almost on any server
d) Secure. It doesn't open any sockets except one required to communicate to MySQL server.
e) Extensible :) Anyone can write additional functionality if required.

Also, because all video conversion requests stored in database table, one can query it to figure out how many requests complete, how many failed, total time of video data to convert etc.

Download text attachment below, edit contents to reflect your actual database/username/directory/program locations, rename it to videoconverter.py or any name your like with extension .py and run.

Any comments and/or suggestions are welcome.

This source code is in public domain and permission granted hereby to use and modify at your taste. The only requirement is to keep original copyright notice in comments.

Install & Use instructions follows in next post

Attached Files



#2 Tigra

Tigra

    Member

  • Members
  • PipPip
  • 17 posts

Posted 01 November 2007 - 10:20 AM

Edit file

DB_HOST = 'your.mysql.serv.er'
DB_DATABASE = 'videoshare'
DB_USER = 'type_your_mysql_username_here'
DB_PASSWORD = 'type_your_password_here'
to your actual database location, name, username (which used to access database) and user password.

NOTE: Because of possible security risk of revealing this information, *NIX users must 'chown' this script to some user ID and 'chmod' it to 700. Windows users have nothing to do with this issue, just keep this script out of reach of hackers.

MPLAYER = '/absolute/path/to/mplayer'
MENCODER = '/absolute/path/to/mencoder'
FFMPEG = '/absolute/path/to/ffmpeg'
FLVTOOL = '/absolute/path/to/flvtool2'
Edit these lines to reflect actual program locations. Windows users please note, that back slash ('\') used as path separator need to be quoted by another backslash, so
'C:\Program Files' need to be written as 'C:\\Program Files'

IN_PATH = '/your/videoin'
OUT_PATH = '/your/videout'
THUMB_PATH = '/your/thumbnail/dir'
TEMP_PATH = '/temporary/dir'
This lines need also to be changed to reflect actual ClipShare video in, flv out, thumbnail and temporary paths. Windows users: read note above, it is applied here too.

Also, script have more adjustable parameters, please read comments to figure out what doing what.

Next step is to make script executable: unix users must chmod +x videoconverter.py, windows users must run it thru 'python.exe videoconverter.py' command

Look at console log messages to debug possible problems.

#3 Tigra

Tigra

    Member

  • Members
  • PipPip
  • 17 posts

Posted 01 November 2007 - 10:22 AM

And, finally, you need to edit a 'upload.php' script. Comment out
$cmd = "$config[phppath] $config[BASE_DIR]\\convert.php $vdoname $vid $ff";
exec($cmd);
lines and right below it write new ones:
$sql = "INSERT INTO conversion_queue SET ts=NOW(), src='$vdoname', dst='$vid', length=$duration";
$conn->execute($sql);
voila!

#4 Tigra

Tigra

    Member

  • Members
  • PipPip
  • 17 posts

Posted 01 November 2007 - 10:34 AM

TODO:
- add command line switches to control script behaviour
- add some more logging regarding MySQL errors
- migrate out of MySQLdb module to standard Python database interface

#5 14m3r

14m3r

    Member

  • Members
  • Pip
  • 3 posts

Posted 01 November 2007 - 01:00 PM

excellent job!
imo, is much more graceful than the original 8)

#6 GodIsLove

GodIsLove

    Member

  • Customers
  • PipPipPip
  • 65 posts

Posted 01 November 2007 - 02:51 PM

Since i'm not the one who host my site, how can I fix my which is hosted in a hosting. ??

#7 Tigra

Tigra

    Member

  • Members
  • PipPip
  • 17 posts

Posted 01 November 2007 - 06:02 PM

First, ask hoster to install Python (if not installed), then follow instructions :)

#8 Darkwoods

Darkwoods

    Member

  • Customers
  • PipPipPip
  • 163 posts

Posted 04 November 2007 - 12:25 AM

if this one is much better from the one on CS why dont they do one similar for CS

#9 14m3r

14m3r

    Member

  • Members
  • Pip
  • 3 posts

Posted 04 November 2007 - 05:59 PM

if this one is much better from the one on CS why dont they do one similar for CS

I think, it needs to be asked the developer. Tigra was simply given with the code, which to me personally seems to more graceful. If to look more closely, in a code of a file upload.php, there was vulnerability (SQL Injection). He has fix it ;]

#10 Darkwoods

Darkwoods

    Member

  • Customers
  • PipPipPip
  • 163 posts

Posted 04 November 2007 - 06:23 PM

I think, it needs to be asked the developer. Tigra was simply given with the code, which to me personally seems to more graceful. If to look more closely, in a code of a file upload.php, there was vulnerability (SQL Injection). He has fix it ;]


yeah thats why i am asking because many CS customer including my self are having problems with convert.php

and CS Staff should do something about fixing this bug

#11 Tigra

Tigra

    Member

  • Members
  • PipPip
  • 17 posts

Posted 07 November 2007 - 10:17 AM

Is anyone interested with this script? Shall I continue work to improve it?

#12 bman

bman

    Member

  • Customers
  • PipPipPip
  • 83 posts

Posted 07 November 2007 - 10:55 AM

i think we need a better how-to on how to install it

#13 zennode

zennode

    Member

  • Customers
  • PipPipPip
  • 76 posts

Posted 07 November 2007 - 04:05 PM

Tigra, I am definitely interested. I am just new ot all this and do not know how to install exactly.

#14 bulforce

bulforce

    Member

  • Members
  • Pip
  • 7 posts

Posted 08 November 2007 - 05:12 PM

I guess the install instructions are not valid for cs 2.5....
does anyone know how to integrate in cs 2.5

thank you

#15 alter_knacker

alter_knacker

    Member

  • Customers
  • PipPipPip
  • 539 posts

Posted 11 November 2007 - 06:22 PM

Is anyone interested with this script? Shall I continue work to improve it?


because of my problem with the thumbs not created i tried your script but then i get the old error "video conversion in progress" instead of "thumbnail not available" and the video itself is looooaaaaading.....

#16 Tigra

Tigra

    Member

  • Members
  • PipPip
  • 17 posts

Posted 14 November 2007 - 12:05 PM

Tigra, I am definitely interested. I am just new ot all this and do not know how to install exactly.

First, sorry for a long AFK. Definitely, i've post an install instruction with next two days. Work in progress.

I guess the install instructions are not valid for cs 2.5....
does anyone know how to integrate in cs 2.5

It's quite simple. Just find place in 'comment.php' script where generic 'convert.php' script is called, comment it out and insert this code instead of commented:
$duration = 0;
$sql = "INSERT INTO conversion_queue SET ts=NOW(), src='$vdoname', dst='$vid', length=$duration";
$conn->execute($sql);
$duration variable is not important for the script, so you can safely pass zero as its value. It is intended only for future scripts, which can give you total time of files in conversion queue.

#17 Tigra

Tigra

    Member

  • Members
  • PipPip
  • 17 posts

Posted 14 November 2007 - 12:07 PM

because of my problem with the thumbs not created i tried your script but then i get the old error "video conversion in progress" instead of "thumbnail not available" and the video itself is looooaaaaading.....

Can you post here output messages of videoconverter.py script regarding any videeo without thumbnails?

#18 Tigra

Tigra

    Member

  • Members
  • PipPip
  • 17 posts

Posted 14 November 2007 - 12:32 PM

Update: for all that have troubles, i post a new, updated version of script.

Please note that script now can be configured via command line switches: run "videoconverter.py --help" to learn more. Default values as usually can be edited by changing script text. Some switches not implemented yet, sorry :)

Attached Files



#19 Tigra

Tigra

    Member

  • Members
  • PipPip
  • 17 posts

Posted 14 November 2007 - 12:37 PM

because of my problem with the thumbs not created i tried your script but then i get the old error "video conversion in progress" instead of "thumbnail not available" and the video itself is looooaaaaading.....

Do you run ClipShare under Windows or Unix?

#20 Tigra

Tigra

    Member

  • Members
  • PipPip
  • 17 posts

Posted 14 November 2007 - 01:35 PM

==============================================================================
Videoconverter script installation routine for ClipShare™ versions 2.0 or
higher
==============================================================================

INTRODUCTION

Videoconverter is a replacement for built-in video conversion routine used
in ClipShare™ media sharing software version 2.0 or higher

COPYRIGHT

This code is in public domain. Permission is granted by an author(s) to
distribute it free of charge and modify to your taste. The only requirement
is to keep original copyright message in script comment lines.

THIS SOFTWARE IS DISTRIBUTED ON AS-IS BASIS WITHOUT ANY WARRANTY. USE IT
ON YOUR OWN RISK. AUTHORS IS NOT RESPONSIBLE OF ANY DAMAGE, DIRECT OR
INDIRECT CAUSED BY USE OR IMPOSSIBILITY OF USE OF THIS SOFTWARE.

REQUIREMENTS

First of all, this script requires Python scripting language version 2.4
or higher installed on your system. You can get Python distributions at

http://www.python.org/download/

page. Windows users must download Windows Installer, which do any
necessary actions to prepare your system for Python use, Unix users can
either download source tarball and build it, or install it from ports
(hello, FreeBSD users! :)) or from other location, such as RPM archive
etc.

If you unsure, is Python installed on your system, or not search your
filesystem for 'python.exe' file, usually located at 'C:\PythonXY'
directory, where X and Y is version major and minor numbers (Windows users)
or execute command 'which python' (Unix users) and look for its output.
If it shows something like '/usr/local/bin/python' than all is ok and you
already have Python installed.

Script relies on several external programs to work properly. This programs
usually will be already installed, because it is required by ClipShare™
software, but for clarity, they are listed below:

- mplayer/mencoder package, used to transcode your videos.
URL: http://www.mplayerhq.hu

- ffmpeg
URL: http://ffmpeg.mplayerhq.hu/

- flvtool, program used to update .flv video metatags.
URL: http://rubyforge.org...jects/flvtool2/

Unix users: please note that flvtool required Ruby programming language
installed.

INSTALLATION

Unix boxes:

1. Download this script, rename it to 'videoconverter' and copy into
/usr/bin or /usr/local/bin directory depending on your system. FreeBSD
users must prefer /usr/local/bin location as it conforms to standard
filesystem layout.

2. run following commands as root user (# is a command prompt in
superuser mode):

# cd /usr/local/bin (or /usr/bin)
# chmod 750 videoconverter
# chown root:wheel videoconverter

This commands sets -rwxr-x--- mode to the script, allowing only
script owner (root) to execute it and disallowing group 'others'
any access to script body, thus preventing leaking any sensitive
information such as database server address, username, password,
etc.

If you absolutely sure about your system security, you can skip
this step.

3. Make this script starting at system boot.

There were several ways to do this, depending on your system. The
simplest way is to write shell script that starts video converter
with following contents (replace <args> with actual script
arguments which can be found by running 'videoconverter --help'):

#!/bin/sh
nohup /usr/bin/videoconverter <args> &

save this text as 'videoconverter.sh' within directory containing
your system startup scripts (usually /etc/rc.d or
/usr/local/etc/rc.d)

4. That's all. Run shell script and videoconverter will be started.

Windows boxes:

1. Download this script and place into some directory. This can be,
for example, C:\Program Files\videoconverter, or C:\videoconverter
Don't forget to add extension '.py' to filename (if not done already)

2. Run 'regedt32.exe': Choose 'Start' -> 'Run', type 'regedt32' and
press Enter.

3. Navigate to the registry folder

'HKLM\Software\Microsoft\Windows\CurrentVersion\Run' and

create string value with name 'VideoConverter' and value
'<pythonpath>\python.exe <path_to_videoconverter> <options>'

where <pythonpath> need to be replaced by actual installation path
of Python, <path_to_videoconverter> to full path where you place
script and <options> to script options.

4. To start videoconverter immediately you may run command above from
'Start' -> 'Run' dialog box.

INTEGRATING WITH CLIPSHARE

Integrating this script share is quite simple. Open 'upload.php' script in
any editor, then find and comment out line like this:

$cmd = "$config[phppath] $config[BASE_DIR]/convert.php $vdoname $vid $ff";
exec($cmd);

then right under commented code add:

$sql = "INSERT INTO conversion_queue SET ts=NOW(), src='$vdoname', dst='$vid', length=0";
$conn->execute($sql);

Voila. Now instead of running generic 'convert.php' script, your ClipShare
software will add a new task into conversion queue where videoconverter is
awaiting for a new jobs.

TROUBLESHOOTING

When experiencing problems, first try to run this script manually from
command line, adding -v DEBUG -l STDERR to its arguments. This will force
script to output some debug info which can be helpful when figuring out what
to do next.

CONFIGURING SCRIPT

Script have many configuration options, which can be tuned for best
performance. You can either edit script by yourself to set reasonable
defaults or pass it via command line switches.

Open videoconverter.py in your favorite editor and read through comments
to figure out what is doing what. Edit and save.

KNOWN BUGS AND ISSUES

- Under Windows, there was known bug with MPlayer, in -vo jpeg:outdir=xxx
command line option: mplayer doesn't accept a drive letter in destination
path, thus failing thumbnail generation routine. To work around this issue,
simply create temporary directory (\TMP or \TEMP) on drive from videoconverter
was run and specify this directory as temporary location using --temp-path=
command line switch.

Please report further bugs or suggestions to tigra@an-net.ru email address

AUTHORS AND CONTACTS

Alexey (Tigra) Skoufyin
mailto:tigra@an-net.ru





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users