8000 Generic version of src_inc_gpc with key parameter · Issue #2423 · haproxy/haproxy · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Generic version of src_inc_gpc with key parameter #2423

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
phihos opened this issue Jan 24, 2024 · 3 comments
Closed

Generic version of src_inc_gpc with key parameter #2423

phihos opened this issue Jan 24, 2024 · 3 comments
Labels
status: fixed This issue is a now-fixed bug. type: feature This issue describes a feature request / wishlist.

Comments

@phihos
Copy link
Contributor
phihos commented Jan 24, 2024

Your Feature Request

Hi,

the fetch method src_inc_gpc can be used to put an entry for the source IP address of the current request into a stick table if a condition is met. The docs show the following example:

acl abuse src_http_req_rate gt 10
acl kill  src_inc_gpc0 gt 0
tcp-request connection reject if abuse kill

Now let me modify this a little bit to show my use-case:

acl abuse src_http_req_rate gt 10
acl kill  src_inc_gpc0(blocked-clients) gt 0
acl already_killed src_get_gpc0(blocked-clients) gt 0
tcp-request connection reject if already_killed
tcp-request connection reject if abuse kill

This is almost functionally the same but the counter has been moved to another table and the expiry of that table determines how long the client will be rejected. The advantage is that the number of entries in blocked-clients reflects the number of currently blocked unique IP addresses. Since the number of entries is visible in the Prometheus metrics generated by HAProxy we can have a nice graph about the number of currently blocked clients.

This works well when the key of the stick table is an IPv4/6 address. But as far is I know there is no way to have an equivalent of src_inc_gpc if the tables from the example had an API key as table keys. The key feature of src_inc_gpc in this example is to only create the entry in the blocked-clients table if the client has been flagged a abusive.

So given that there is a variable txn.api_key something like this would be nice:

track-sc0 var(txn.api_key)
#...
acl abuse src_http_req_rate gt 10
acl kill  inc_gpc0(var(txn.api_key), blocked-clients) gt 0
acl already_killed get_gpc0(var(txn.api_key), blocked-clients) gt 0
tcp-request connection reject if already_killed
tcp-request connection reject if abuse kill

What are you trying to do?

Implement rate-limiting with something different that a source IP address as key.

Output of haproxy -vv

HAProxy version 2.9.3-1ppa1~jammy 2024/01/18 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.9.3.html
Running on: Linux 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -O2 -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_SYSTEMD=1 USE_OT=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL +OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=128).
Built with OpenSSL version : OpenSSL 3.0.2 15 Mar 2022
Running on OpenSSL version : OpenSSL 3.0.2 15 Mar 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with Lua version : Lua 5.3.6
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with OpenTracing support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.39 2021-10-29
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 11.4.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=

Available services : prometheus-exporter
Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] compression
	[FCGI] fcgi-app
	[  OT] opentracing
	[SPOE] spoe
	[TRACE] trace
@phihos phihos added the type: feature This issue describes a feature request / wishlist. label Jan 24, 2024
@Darlelet
Copy link
Contributor
Darlelet commented Jan 26, 2024 8000

Interesting use-case, thanks for sharing it :)

Despite most sticky counters actions and fetches likesc_get_gpc and sc_inc_gpc being compatible with generic key matching thanks to track-sc* action... it's true that the track-sc* action won't help here: as you pointed out, you want to be able to check for an existing entry WITHOUT creating it if it doesn't exist (which is what you already achieve thanks to src_get_gpc0 + conditional src_inc_gpc0 combination, yet using track-sc action to leverage generic matching key would automatically populate missing entries)

I can't tell if implementing a whole new set of generic fetches based on src_inc_gpc0 and friends (that would take the key as argument as you suggested) would be the most convenient solution.

But maybe providing an optional argument to track-sc to control whether the action is allowed to create the entry or not could help?

ie:

http-request track-sc0(0) var(txn.api_key) table blocked-clients #  don't create if not found
acl already_killed sc_get_gpc0(0) gt 0
http-request reject if already_killed
http-request track-sc0 var(txn.api_key) table blocked-clients # make sure to create the entry if missing
acl abuse src_http_req_rate gt 10
acl kill  sc_inc_gpc0(0) gt 0
http-request reject if abuse kill

Although it seems this can already be achieved (in a not so convenient way) by making use of the optional <table> argument that most sticky counters fetches support (to look for the same key as the one currently tracked, but in another table), ie: by sacrificing an intermediate table on which the initial track-sc with the desired key will be performed.

something like that (not tested though, I might be wrong):

http-request track-sc0 var(txn.api_key) table temp-clients
acl already_killed sc_get_gpc0(0, blocked-clients) gt 0
http-request reject if already_killed
acl abuse src_http_req_rate gt 10
acl kill  sc_inc_gpc0(0, blocked-clients) gt 0
http-request reject if abuse kill

Edit: forget about this last example, it is not a viable workaround: sc_inc_gpc0(0, blocked-clients) won't create the entry if it doesn't exist, so we're back to our initial issue (specifying an optional table to sample fetches: does simple lookup so it fails if the entry is missing)

@wtarreau
Copy link
Member

I think that what Philipp would like in fact, is that we'd provide a split version of src_inc_gpc0(): that one combines two operations at one, a sample fetch function (src) and a converter (inc_gpc0). I don't see any reason why we couldn't have a set of get_gpc0(), inc_gpc0() and clr_gpc0() as converters that apply on the input sample. In fact there's already table_gpc0() which does the get_gpc0() one. We'd in fact need to implement table_inc_gpc*() and table_clr_gpc*(). I don't see any technical showstopper against this and it makes a lot of sense to me. It would even allow to simplify the doc by referring to the canonical converters everywhere ;-)

@wtarreau wtarreau added the status: reviewed This issue was reviewed. A fix is required. label Jan 26, 2024
haproxy-mirror pushed a commit that referenced this issue Jan 16, 2025
As discussed in GH #2423, there are some cases where src_{inc,clr}_gpc*
is not sufficient because we need to perform the lookup on a specific
key. Indeed, just like we did in e642916 ("MEDIUM: stktable: leverage
smp_fetch_* helpers from sample conv"), we can easily implement new
table converters based on existing fetches. This is what we do in
this patch.

Also the doc was updated so that src_{inc,clr}_gpc* fetches now point to
their generic equivalent table_{inc,clr}_gpc*. Indeed, src_{inc,clr}_gpc*
are simply aliases.

This should fix GH #2423.
@Darlelet Darlelet added status: fixed This issue is a now-fixed bug. and removed status: reviewed This issue was reviewed. A fix is required. labels Jan 16, 2025
@Darlelet
Copy link
Contributor

@phihos @wtarreau it should be ok now with table_inc_gpc* and table_clr_gpc* generic converters

@capflam capflam closed this as completed Apr 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: fixed This issue is a now-fixed bug. type: feature This issue describes a feature request / wishlist.
Projects
None yet
Development

No branches or pull requests

4 participants
0