Releases: Lookyloo/lookyloo
Releases · Lookyloo/lookyloo
v1.31.0
This release contains the changes and bugfixes we worked on during GeekWeek X, we'd like to thank the CCCS for having us, and organizing this event.
New Features
/json/advanced_search: Returns Captures matching multiple parameters (IP, Hostname, URL, Hash) by @GavinMacNabb- Docker compose files for dev and prod by @litobro
- New module to automatically submit URLs from Lookyloo to CCCS AssemblyLine by @litobro
/json/ip_info: Returns the captures containing a specific IP address/json/<string:capture_uuid>/ips: Returns all the IPs contained in a capture/json/favicon_info: Returns the captures containing a specific favicon/json/<string:capture_uuid>/favicons: Returns all the favicons in a capture.- Note: playwright doesn't download the favicon, so we fetch them after the fact by extracting the possible favicons from the rendered HTML. This call may return more than one favicon.
/json/hash_info: Returns the captures containing a resource with a specific hash (sha512)- Normalize the JSON API endpoints to they support pagination, accept the same parameters, and return data in the same format
Bugfixes
- Multiple calls to get captures from specific indicators (Hostnames, URLs, IPs, body hashes ...) were not working properly when queried via the API
- Many improvements in error handling
- When the capture queue is really long and they're not captured within one hour (by default), they were expired on Lacus side. Now, if it happens, but the capture is still in the queue on Lookyloo side, it will be re-queued into Lacus.
What's Changed
- Add modules folder to docker-compose.yml to allow live development by @litobro in #1065
- FEAT: Support AssemblyLine Submission by @litobro in #1066
- New POST /json/advanced_search endpoint by @GavinMacNabb in #1071
New Contributors
- @Wachizungu made their first contribution in #1053
- @litobro made their first contribution in #1065
- @GavinMacNabb made their first contribution in #1071
Full Changelog: v1.30.0...v1.31.0
Contributors
Wachizungu, litobro, and GavinMacNabb
Assets 2
v1.30.0 - GeekWeek X
New Feature
This release adds a new endpoint /json/remote_lacuses to get a list of all the remote lacus instances configured on the lookyloo instance, if they're up, and all the proxies they expose, support Lacus v1.15.0:
{
"is_up": true,
"name": "default",
"proxies": {
"Netherlands": {
"description": "Proxy for Netherlands",
"meta": {
"provider": "wireguard"
}
},
"Tor": {
"description": "Trigger the capture via the tor network.",
"meta": {
"provider": "Tor Project"
}
}
}
}
Bugfixes and changes
- Allow for much bigger trees by increasing recursion limit
- Improve the way we retry failed captures
- Avoid deadlock on the UI if the monitoring interface is non-responsive
- Avoid exception on the capture page if lacus is temporary unresponsive
- Avoid exception if archives are not on S3
- Avoid race condition between enqueuing and starting the capture.
And all that, just on time for Geekweek X \o/
Full Changelog: v1.29.0...v1.30.0
v1.29.0
New feature
Lacus v1.14.0 supports pre-configured proxies, this release exposes this feature in Lookyloo, as shown below.
Screencast_20250423_222904.webm
In this case, we use Socks5 proxies: tor, and wireproxy with ProtonVPN and their Wireguard VPN export.
Full Changelog: v1.28.1...v1.29.0
v1.28.1 - Hackathon release
New features
-
Get, store, display and submit storage state
-
Modal to display downloaded files, submit to Pandora if possible
- Add support for multiple remote Lacus (WiP to offer multiple uplinks and captures from different countries)
- Allow to search redirects on the index page
- Improve and speedup search on all tables
- Pull capture from another instance
-
Push capture to a remote instance
-
Render cookie name index from storage state
-
Show cookie name frequency in storage modal
-
Mastodon bot: @lookyloo@social.circl.lu
Changes
- Avoid long string to break tables
- Speedup rendering of tables with jinja 2 templates
- Support for kvrocks installed from the deb package
- Initialize monitoring and pandora at use time, allows to start them after lookyloo is started
- Improve archiver shutdown
- Return HTTP 404 on calls against non-existing files
- Compress full archive export
- Update Python dependencies
- Update JS dependencies
Full Changelog: v1.28.0...v1.28.1
v1.28.0
New features
- Support for captures with a headed browser (setting
allow_headed). Only possible if lacus is running in a graphical environment. (see below for details) - Complete rewrite of the indexing system, allows pagination (transparent on upgrade, but may take a while, be careful if you have a big instance)
- Index IPs and TLDs
- Link hostname and IPs indexes, expose it on the UI
- Trigger CIRCL Passive DNS wherever possible (on IPs and on Hostnames)
- Allow to share direct links to IP/Hostnames/Favicon/... modals
- Push capture to another Lookyloo instance (typically, a headed capture done locally to a central repository)
- Major overhaul of the categorization, use MISP Taxonomy by default (dark-web)
- Optionally makes modules admin only (avoid using all the tokens available on 3rd party APIs)
- Enable full text search on indexes when relevant
- Optional auto-submit of onion URLs to an AIL instance
Changes
- MISP export improvements
- Refactoring and improvements in the javascripts, massive speedups
- Render indexes with AJAX
- Render hostnode popup much faster by moving indexes to a subsequent page
- Submit any file to Pandora
- Trigger optional auto-report in background script
- Allow user accessible MISP servers
- Remove RiskIQ module (killed by microsoft)
- Support KV Rocks 2.11 (major speed improvements)
- Many improvements in logging, reduce noise, and use WARNING more appropriately
- Improve favicon rendering
- Update URLHaus module as it now requires an auth key
Fixes
- Re-enable submit to pandora buttons
- Deduplicate notifications
- Many, many other bugfixes
Notes for headed captures
The classical use of Lookyloo is to have it running on a server with no graphical interface (no X/Wayland server). The capture with Playwright uses a headless browser, runs some interactions on the page (see PlaywrightCapture for details), and finishes after a certain amount of time and/or no traffic. This method is good enough most of the time, but all the interactions on the page are predefined and cannot be modified by the user triggering the capture.
If you use a dedicated lacus instance, please refer to the lacus release for details first. And you need the following to trigger a headed capture:
- The configuration setting
"allow_headed" = Trueinconfig/generic.json - Lookyloo installed on a machine with a graphical interface (or at least lacus, if you're using a remote instance)
- [Web UI] On the capture page, in
Capture configuration-> untickUse headless browser(it it's missing, checkconfig/generic.jsonand restart the app) - [Web UI] Optionally, on the capture page, in
Capture configuration-> setMax capture timeto a time, if needed (90s by default) - [API] Pass
headlessset toFalsein the capture settings - [API] Optionally
general_timeout_in_secset to the amount of time you want to interact with the page (it is set to 90 by default)
The new headed capture mode opens a full browser configured with the settings passed to the capture, and none of the predefined interactions. It lets the user interact with the page for a set amount of time (general_timeout_in_sec in the API, or Max capture time on the web interface), stops the capture, and store the result as usual. It is mostly helpful to manually bypass captchas and other techniques used by websites to detect bots.
mondial.webm
PRs
What's Changed
- change redis to valkey by @sebdraven in #1028
New Contributors
- @sebdraven made their first contribution in #1028
Full Changelog: v1.27.0...v1.28.0
Contributors
sebdraven
Assets 2
v1.27.0
This release requires some system upgrades:
- Valkey 8.0+
- Kvrocks 2.10+
- Python 3.9+
Changes
- Improve documentation on capture page
- Speedup on-demand stop of archiver script
- Make all remaining indexes paginated
- Use new indexes to speedup rendering of hostnode popups and all views relying on indexes
- Maintenance and bugfixes
For Lacus, LacusCore and PlaywrightCapture changes, see Lacus release notes.
Full Changelog: v1.26.1...v1.27.0
v1.26.1
New features
- Major improvements in indexes, allows to paginate hits based on capture timestamp
- Add index for TLDs
- Transparent migration of urls, hostnames, ressources, HTTP Headers Hashes,and cookies indexes to new format
- Run many more DNS requests (MX, SOA, NS)
- Use new indexes on web interface, speedup rendering
- Optionally disable JavaScript during capture
What's Changed
- build(deps): bump docker/build-push-action from 5 to 6 by @dependabot in #939
Full Changelog: v1.26.0...v1.26.1
Contributors
dependabot
Assets 2
v1.26.0
Lacus, LacusCore, and PlaywrightCapture changes
- See Lacus v 1.11.0 release notes for details
- Many bugfixes and improvements
- Use more recent browsers
Har2Tree
Full Changelog: Lookyloo/har2tree@v1.25.0...v1.26.0
- Improve handling of embedded content
- Properly generate docs
Lookyloo
Full Changelog: v1.25.0...v1.26.0
- Improve typing for capture settings (Pydantic)
- Make it possible to categorize captures
- Improve error handling, logging
- Index categorized captures
- Get Captures via UUID in the API
- Add controller to Start/Stop scripts individually
- Add CSP HTTP headers whenever possible
What's Changed
- putting the login message in the right block by @adrima01 in #924
- Categories by @AntoniaBK in #925
- new: get uuids by category via API by @adrima01 in #926
- add new endpoint to remove capture by @jeroengui in #929
New Contributors
- @jeroengui made their first contribution in #929
Contributors
jeroengui, adrima01, and AntoniaBK
Assets 2
v1.25.0 - Pass the Salt 2024
New features
- Endpoint to remove capture from the index (safely copies it in a directory)
- Configurable way to gather contact information for takedown of malicious websites
- FuzzImy hash based on HTML structure of the rendered page (algorithm of CERT PL)
- New simple capture page without any of the options
- Optional user config to overwrite capture settings and/or have default parameters
- Download tree as PNG (as long as the tree isn't too big)
- Optional auto-report on capture page (admin only)
- Optionally change the index page to the capture page, useful when the index is very big and takes a long time to load
Changes
- Many improvements when rendering panels on tree
- New menus
- Vast amount of bugfixes and improvements everywhere in the project by @AntoniaBK and @adrima01, see below.
- Support for valkey, new kvrocks, Ubuntu 24.04
- Optionally disable SRI validation while developing
- Partial removal of jQuery
- Many improvements in loading index, uses caching much more efficiently
- Restart webserver more often, avoids memory leaks
- Improve logging, reduces noise
What's Changed
- Lookup of abuse-c by @AntoniaBK in #904
- Remove redundant clause by @AntoniaBK in #906
- adding uwhoisd installation by @adrima01 in #907
- Fix: DataTable rename to treeHashesTable by @AntoniaBK in #909
- adding 3rd party report to mail by @adrima01 in #908
- Menu by @AntoniaBK in #910
- changes so that you can ignore the sri while developing by @adrima01 in #911
- Simple interface by @adrima01 in #912
- fix: removing unnecessary script and jQuery by @adrima01 in #913
- new: downloadable tree as png by @adrima01 in #915
- New: admin-only checkbox for auto-report by @AntoniaBK in #914
- fix: [modules] Gracefully accept no hashlookup fixes #916 by @cvandeplas in #917
- Fixing typo by @adrima01 in #918
- New: upload a capture via the API by @AntoniaBK in #919
- Recent captures by @adrima01 in #921
- Update generic.json.sample by @adrima01 in #923
New Contributors
- @cvandeplas made their first contribution in #917
Full Changelog: v1.24.0...v1.25.0
Contributors
cvandeplas, adrima01, and AntoniaBK
Assets 2
v1.24.0
New features
- Optionally attempt to allow tracking on capture, see lacus v1.9.0 release for details.
- [Admin Only] Index all captures, not only the public ones (in kvrocks instead of redis)
- Multiple improvements in correlation pages (Favicons, ressources
- Index favicons
- Compute favicons MM3H and like it to Shodan
- Search and index captcha IDs (reCaptcha, hCaptcha and Cloudflare)
What's Changed
- Changed misp_url by @adrima01 in #894
- Proxy with VT module by @DocArmoryTech in #897
- Module response added by @adrima01 in #898
- Made send_email available from the API by @AntoniaBK in #899
- Speedup async capture checks when the backlog in long
- Improve favicon rendering on tree
- Split capture building and indexing in two different scripts
- Reduce memory usage for indexing scripts
New Contributors
- @adrima01 made their first contribution in #894
- @DocArmoryTech made their first contribution in #897
- @AntoniaBK made their first contribution in #899
Full Changelog: v1.23.0...v1.24.0
Contributors
DocArmoryTech, adrima01, and AntoniaBK