Twitter bot for generating photo descriptions
Twitter: https://twitter.com/GramtionBot
Source Code: https://github.com/lRomul/gramtion
This repo contains the source code of the Twitter @GramtionBot for generating photo descriptions.
Use cases and intends:
- Help visually impaired Twitter users. Good image descriptions (alt text) will help them understand what is happening in an image. Instagram and Facebook use deep learning for image captioning. Twitter users can only add custom alt text descriptions themselves. Automation of alt text generation will help Twitter be more accessible.
- Collect dataset for image captioning (legal issues for this use case are yet to be discussed). Annotations can be done by creating polls about prediction quality and getting corrected descriptions from users. Twitter API v1.1 has not the ability to create polls, but it will be added in API v2.
Tweet photo with mention @GramtionBot or reply with mention to a tweet with a photo and the bot will send you an auto-generated image description.
Links to example 1 and example 2 tweets.
Gramtion is mainly built from ready-to-use third party libraries:
- Image captioning model taken from self-critical.pytorch.
- Bot written with Tweepy.
- Configuration settings implemented with pydantic.
- Docker image based on Dokai.
- Image labels by Google Vision Ai.
- The model can only handle photos, for other types of images the predictions are quite random.
- Some descriptions may be confusing. Annotations may be created by using polls about prediction quality and getting corrected descriptions from users. Twitter API v1.1 has not the ability to create polls, but it will be added in API v2 endpoint
POST /2/tweets. - Some results may reflect inherent gender and racial biases of open datasets.
- Still waiting for feedback from visually impaired users to add some features or change the interface to make the bot more accessible.
To run an instance of the bot you need to install Docker and create Twitter API auth credentials.
If you have a Twitter developer account, but don't want to use it as a bot account, you can authenticate a new user that’s not has a developer account with twurl.
-
Create
.envfile with credentials.CONSUMER_KEY={{ consumer_key }} CONSUMER_SECRET={{ consumer_secret }} ACCESS_TOKEN={{ access_token }} ACCESS_TOKEN_SECRET={{ access_token_secret }} -
Setup Google Vision AI and create account key (link). Copy key as
google_key.json. -
Run Docker container with running the bot
docker run -d --restart=always \ --env-file .env \ -v google_key.json:/workdir/google_key.json \ --name=gramtion \ ghcr.io/lromul/gramtion:0.0.3 -
Open logs
docker logs -f gramtion
-
Stop container
docker stop gramtion docker rm gramtion