Creating custom Queue and cloud module

DannyA
@dannya
9 years ago
584 posts

Can someone explain how to create a module that uses worker queues that can run a command line tool on a conversion server?
updated by @dannya: 03/30/15 03:25:36AM

@brian
9 years ago
10,136 posts

There's really nothing special about a module that uses queues - basically you:

- register the name of the function that will be run when a queue entry is received that matches the queue - for example:

jrCore_register_queue_worker('jrAudio', 'audio_conversions', 'jrAudio_convert_file', 0, 1);

This tells the core:

- when a new queue entry comes in to the "audio_conversions" queue
- call the "jrAudio_convert_file" function and pass it $_queue - $_queue contains all the data that was passed in when the queue was created
- "0" means there is NO LIMIT to the number of queue entries this function will process in a row (i.e. if you set it to "1" it would process 1 queue entry then exit).
- the "1" says "only allow one function of this type to be running at the same time".

From the worker function:

- if you return TRUE the queue entry is deleted (that's how you tell the core you were succesful and the queue entry can now be removed).
- if you return FALSE then the queue goes back into the stack for another worker to pickup
- if you return a NUMBER it tells the core to "sleep" the queue for that many seconds - i.e. if you did:

return 3600;

it would put the queue entry back on the stack but not allow any worker to pick it up for 3600 seconds.

So basically you just need to register the queue worker in the module's init function and that's it.

Let me know if that helps.

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net
updated by @brian: 02/24/15 11:45:06AM

DannyA
@dannya
9 years ago
584 posts

So how does the function on the remote server, or even on the main app server, know to look at the queue for a job? And how do you set it to get a file from s3.

Let me give you the specific use case.

I have a CLI for a waveform creation tool. I want the waveform creation to be performed by one of my conversion worker servers.

1. I assume the bpm detection module has to be on the conversion server.
2. I assume, when a file is finished uploading, I create a new queue task " create waveform".
3. How does the waveform creation module know to look at the queue for a new job?
4. How does the waveform creation module know to look in s3 for a specific file instead of local file system
5. How do i tell the waveform module to write the resulting image to s3 instead of the local file system?

@michael
9 years ago
7,692 posts

I've added this discussion to the docs:

"The Queueing System"
https://www.jamroom.net/the-jamroom-network/documentation/development/1543/the-queueing-system

@brian
9 years ago
10,136 posts

DannyA:
So how does the function on the remote server, or even on the main app server, know to look at the queue for a job?

It does not need to - the core handles it.

Quote:
And how do you set it to get a file from s3.

If you are using the S3 module, and it is the active media system, you can just use jrCore_media_read_file(), which will "read" a file from S3 and bring it over.

Quote:
Let me give you the specific use case.

I have a CLI for a waveform creation tool. I want the waveform creation to be performed by one of my conversion worker servers.

1. I assume the bpm detection module has to be on the conversion server.
2. I assume, when a file is finished uploading, I create a new queue task " create waveform".

Yes to both...

Quote:
3. How does the waveform creation module know to look at the queue for a new job?

It doesn't:

- when the queue is CREATED, it will be created on the same server OR on the queue server (if you have configured a queue server);
- when the core on the server that has the queue entry "sees" there is a queue entry, it will fire up the right function.

So you don't need to worry about that.

Quote:
4. How does the waveform creation module know to look in s3 for a specific file instead of local file system

It doesn't need to - as long as all the file reading/writing/checking is done using the jrCore media functions - i.e.:

if (jrCore_medie_file_exists(1, 'MyModule_1_file.mp3')) {

    // somefile.mp3 exists - either locally OR on S3 (if using S3)
    $_item = jrCore_db_get_item('MyModule', 1);  // get DS info for item id 1

    // If the file is on S3 it will copy it to /tmp/file.mp3 - or wherever
    jrCore_confirm_media_file_is_local(1, 'MyModule_1_file.mp3', $_item, '/tmp/file.mp3');
}

So what this does is:

- checks to see if the file 'MyModule_1_file.mp3' exists for profile_id 1
- if it DOES exists, it ensures there is a local copy of it at /tmp/file.mp3

You don't have to care what file system the file is actually on.

When done, you can "write" the file back:

jrCore_write_media_file(1, 'MyModule_1_alternate_file.mp3', '/tmp/file.mp3');

This would write the /tmp/file.mp3 file out to the proper location (and call it MyModule_1_alternate_file.mp3) regardless if you are using the local file system, s3, whatever.

The idea here is that as long as you avoid using function calls like is_file(), file_exists(), etc. (basically functions that only work with a local file) you don't have to care - the jrCore media functions will "wrap" everything and basically abstract the filesystem away.

Let me know if that helps.

--
Brian Johnson
Founder and Lead Developer - Jamroom
https://www.jamroom.net

Creating custom Queue and cloud module

Tags