An HTTP job processing service.
Here the definition of a job is a collection of tasks, where each task has a name and a shell command. Tasks may depend on other tasks and require that those are executed beforehand. This service takes care of sorting the tasks to create a proper execution order.
{
"tasks":[
{
"name":"task-1",
"command":"touch /tmp/file1"
},
{
"name":"task-2",
"command":"cat /tmp/file1",
"requires":[
"task-3"
]
},
{
"name":"task-3",
"command":"echo 'Hello World!' > /tmp/file1",
"requires":[
"task-1"
]
},
{
"name":"task-4",
"command":"rm /tmp/file1",
"requires":[
"task-2",
"task-3"
]
}
]
}The server can respond in two format: JSON and bash script in the form of text.
The content will be negotiated via Accept header:
Accept: application/jsonwill give out a json response.Accept: text/plainwill respond with the bash script in plain text.
The above example will be processed to this response:
[
{
"name":"task-1",
"command":"touch /tmp/file1"
},
{
"name":"task-3",
"command":"echo 'Hello World!' > /tmp/file1"
},
{
"name":"task-2",
"command":"cat /tmp/file1"
},
{
"name":"task-4",
"command":"rm /tmp/file1"
}
]The bash script response will look like this:
#!/usr/bin/env bash
touch /tmp/file1
echo "Hello World!" > /tmp/file1
cat /tmp/file1
rm /tmp/file1This way, it's possible to run it directly like so:
$ curl -d @mytasks.json -X POST -H 'Accept: text/plain' -H "Content-Type:application/json" http://localhost:4000/process_job | bashRequirements
- elixir 1.8
- erlang 21.2
To start your Phoenix server:
- Install dependencies with
mix deps.get - Create and migrate your database with
mix ecto.setup - Start Phoenix endpoint with
mix phx.server
- You can open production Swagger API documentation at https://job-processor.gigalixirapp.com/api/swagger.
- You can re-generate or update it by running
mix phx.swagger.generate - locally will be available at
http://localhost:4000/api/swaggeronce phoenix is running.
- Run tests with
mix test - Run coverage with
mix coveralls - Codecov is being used as a coverage webpage.
Credo is a static code analysis tool for the Elixir language with a focus on teaching and code consistency.
- Run credo with
mix credo
Dialyzer can analyze code with typespec to find type inconsistencies and possible bugs.
- Run dialyzer with
mix dialyzer
Be aware this takes a while to run for the first time. Runs after the first will be a lot faster.
Code documentation generated with ex_doc:
mix docs && open doc/index.html
The project is being built and tested in Circle CI.
Using Distillery:
mix releaseto generate release.
Currently being deployed to production at gigalixir:
I made the assumption that authentication, authorization and persistence are outside the scope of this test.
For functionality exposed via HTTP, Phoenix seemed like a good choice. It has great productivity, facilitators, code generators and much more at the cost of very little overhead. I started a new phoenix app without frontend and also without Ecto, because of the above assumptions.
The problem of ordering tasks seemed to me to be in the domain of Graphs. Since elixir doesn't have this data structure native, I used a library.
Then, the basic idea was to build a directed acyclic graph, and to do a breadth-first scan to find all necessary tasks in an order that makes sense.
It looks something like this:
