stage0: attach to the app's stdin/stdout/stderr #1799

yifan-gu · 2015-11-26T01:29:01Z

Currently, kubernetes provides the ability to attach to the app's stdin/stdout/stderr using docker attach. It is mainly for debugging purpose.
So in order to complete the kubernetes' requirement, we would need to implement something similar, possibly by implementing in rkt enter, e.g. rkt enter --attach stdin --attach stdout?

Any ideas @vcaputo @jonboulle ?

The text was updated successfully, but these errors were encountered:

vcaputo · 2015-11-26T08:02:00Z

So you're looking for something analogous to a screen/tmux attach to the pty of the running app?

I don't think systemd implements that functionality today, but I could be wrong.

alban · 2015-11-30T17:05:06Z

Can Kubernetes get the past stdout as well, or are we only interested in the stdout from the moment we attach to the app?

alban · 2015-11-30T17:11:27Z

I also don't think systemd implements that functionality. I asked on systemd/systemd#2069.

yifan-gu · 2015-11-30T18:39:10Z

Can Kubernetes get the past stdout as well, or are we only interested in the stdout from the moment we attach to the app?

@alban The docker attach command itself doesn't return the past log, but the API does, it as an option to control this logs – 1/True/true or 0/False/false, return logs. Default false.
https://docs.docker.com/engine/reference/api/docker_remote_api_v1.19/

euank · 2016-09-09T20:19:49Z

So, option 1 of TTYPath above.. I did some quick prototyping:

I ran this perl script:

#!/usr/bin/perl
use IO::Pty;
use Fcntl;

my $pty = new IO::Pty;
fcntl $pty, F_SETFD, 0; # clear close-on-exec
system "urxvt -pty-fd " . (fileno $pty) . "&";
close $pty;
printf("tty: slave %s, master %s\n", $pty->slave->ttyname(), $pty->ttyname());

sleep 999

And then this systemd unit:

[Unit]
Description=attach-test

[Service]
ExecStart=/bin/sh
StandardInput=tty-force
StandardOutput=tty
StandardError=tty
TTYPath=/dev/pts/$ptsFromPerl
TTYReset=yes
MountFlags=shared
RootDirectory=/home/esk/alpine-root

Before starting the unit, I created a bindmount for my pts over to my alpine-root. I also had to fiddle with my proc mount.

That gave me what I expected; a running 'sh' process which the new perl-spawned urxvt could talk to, where the tty command agreed with the TTYPath I gave.

I propose that from there, there follow two ways to implement option 1:

User-heavy manual attach

We could allow a user to specify effectively use-tty=/path/to/pts/on/host for each application in a pod. The above options would be set and the provided pts would end up getting bindmounted all the way through to the application as well.

It would then be up to the user to drain, manage, and attach using this pts.

We would document that the use-tty flag will result in logs not being written to the journal for that applicatin's stdout/stderr as well.

First-class rkt attach

We could also create a more first class integration where rkt adds a --attachable flag for applications. When this flag is set, on any application, a single additional process per pod is created which makes and manages a pty (say stage1-pty-helper). The pty created by that application is then passed to the application.

The stage1-pty-helper provides an interface that allows a new first-class rkt command rkt attach to work, and the helper could also write logs to the journal on the application's behalf, thus preserving something very close to the existing logging behaviour.

The first of these options is more flexible, but will not be generally usable by a cli-user, and will also require any system that does with to use it to do some heavy lifting. I think either option is reasonable.. but most of all I think we should decide on one of the two to go with it so long as we can all agree there isn't a better option and there isn't a problem with this one.

I know @yifan-gu was looking at some prototyping related to this stuff, so I'd like him to throw in an opinion, as well as @lucab, and if both of you can agree on a specific approach, let's get going on implementing. @lucab, if you can figure out if @alepuccetti is the right person to own this or if there's a better person, that would be awesome!

euank · 2016-09-09T20:24:56Z

@lucab missed your comment for a second: my take on your options

That's what I'm leaning towards above, we might be able to do 1 + helper that writes to journald to end up in an okay place. I don't think this needs a new stage1 if we gate it with an --attachable flag or such.
That seems like a better long-term solution if systemd thinks it's a good idea! Unfortunately, if systemd is connected to the tty for logs, systemd would also need to implement the tooling to allow attaching to that tty. I think this was ruled out in the systemd issue above, but we could raise it again for this. I'd rather go with 1 in parallel while we figure out if we can do this one, if this is the preferred option
Sounds similar to the question raised here, which I don't know that we're confident in an answer to. I think this will be more fragile and less attractive than 1.

Sorry for missing your comment for a bit and adding a giant wall of text that duplicates a chunk of what you said!

lucab · 2016-09-13T07:52:52Z

@euank thanks for the feedback. I'm going with option 1, while leaving an open window for option 2 for later (once we have a working PoC on our side we can re-discuss systemd details). I'm dropping option 3 for the moment.

I'm starting from stage1/stage2 internals, hoping to have something crude and working at the low level. Once there, I'll gather some more feedback before implementing host integration (k8s and stage0) which also includes a decision on --attachable.

lucab · 2016-09-19T13:48:46Z

I have a very crude ttymux service (based on TTYPath= above) working in my iottymux branch, which just exposes the pty raw binary stream over a network socket for stage0 consumption. Out-of-band signaling (eg. for resizing) is not implemented at the moment. This is effectively quite similar to a per-app tty telnet service.

However, it looks like the requirements from kubernetes are much more complex and require being able to attach either to a tty (ie. a single in/out stream plus a control back-channel) or to every single stream separetely (ie. separate stdin/stdout/stderr channels). See kubernetes/kubernetes#32869 for all the details.

For this reason, it looks like my current PoC is not enough and we need support from systemd to be able to handle streams separately and without a tty. I just submitted a proposed implementation of this to systemd (systemd/systemd#4179).

euank · 2016-09-26T23:18:19Z

I'm confused as to why those systemd changes are needed. Why can't the iottymux program just speak a different protocol that is sufficiently rich to support everything?
At worst/messiest, it seems like you could just have a bind-mounted directory that it writes ttys to which are then consumed by the external user however they seem fit.

Mind clarifying why additional iottymux code or tweaking its interface doesn't allow expressing multiple streams?

lucab · 2016-09-27T08:10:52Z

@euank the problem is not between k8s and iottymux, but between iottymux and single apps (as systemd services). When running with a TTYPath= applications are attached to a pty, which means that there are terminal semantics (eg. resizing, echo, etc.) and that all output streams (stdout/stderr) are muxed together. As iottymux is reading from pty fd, it won't be able to distinguish stdout from stderr. This is similar to docker -t option.

Systemd patching is needed in order to provide separate stdin/stdout/stderr streams to iottymux (and without any tty semantics in the middle).

philips · 2016-10-28T04:05:16Z

What is the next step on this?

lucab · 2016-11-03T08:58:59Z

Systemd implementation is done and merged into systemd git (master, post v231): systemd/systemd#4179

According to plans, systemd v232 should have already been released but unfortunately this release seems to be taking longer than expected, so I just asked for an ad-hoc backport in order to have this available in stage1: coreos/bugs#1645

I've ported my initial PoC to a proper golang stage1 sidecar, and added a rkt attach subcommand using the new entrypoint for attaching. It is currently available in a private branch of mine at https://github.com/lucab/rkt/tree/wip/lucab/sandbox-iottymux (building this requires stage1-src flavor from current systemd master).

This is already rebased on top of the current CRI rework (#3318), which should land into master in this release cycle. It was already briefly demoed in the latest community sync, see recording.

Next steps from here are:

Land the CRI rework and a backported systemd in master
Rebase the iottymux branch on top of it, cleanup the code
Add tests and documentation

lucab · 2016-11-23T10:04:14Z

All blockers for this should be now gone. Tracking implementation progress at #3396.

yifan-gu added kind/enhancement area/usability priority/P2 labels Nov 26, 2015

jonboulle added the technology/kubernetes label Nov 26, 2015

iaguis added this to the v0.13.0 milestone Nov 27, 2015

alban mentioned this issue Nov 30, 2015

attach stdin/stdout to a running service systemd/systemd#2069

Closed

jonboulle added the help wanted label Dec 2, 2015

iaguis added priority/P1 and removed priority/P2 labels Dec 4, 2015

alban modified the milestones: v0.15.0, v0.14.0, v0.16.0 Dec 18, 2015

alban modified the milestones: vfuture, v0.16.0 Jan 14, 2016

alban added priority/P2 and removed priority/P1 labels Jan 14, 2016

alban modified the milestones: v1+, vfuture Jan 15, 2016

jonboulle modified the milestones: v1.2.0, v1+ Feb 4, 2016

iaguis modified the milestones: v1.3.0, v1.2.0 Mar 18, 2016

philips mentioned this issue Mar 22, 2016

API: use cases for kubectl attach? kubernetes/kubernetes#23335

Closed

jonboulle added priority/P1 and removed priority/P2 labels Mar 22, 2016

alepuccetti removed their assignment Sep 12, 2016

lucab self-assigned this Sep 13, 2016

lucab modified the milestones: v1.16.0, v1+ Sep 13, 2016

s-urbaniak added the area/cri label Sep 15, 2016

lucab mentioned this issue Sep 19, 2016

core/exec: add a named-descriptor option ("fd") for streams systemd/systemd#4179

Merged

lucab added the depends-on/systemd label Sep 29, 2016

lucab modified the milestones: v1.18.0, v1.16.0 Sep 29, 2016

squeed modified the milestones: v1.19.0, v1.18.0 Oct 27, 2016

lucab mentioned this issue Nov 3, 2016

systemd: please backport streaming-fd patch (systemd PR 4179) coreos/bugs#1645

Closed

lucab mentioned this issue Nov 7, 2016

Implementing rkt app add and rkt app rm #3136

Closed

1 task

lucab modified the milestones: v1.20.0, v1.19.0 Nov 9, 2016

This was referenced Nov 18, 2016

stage1: update coreos base to alpha 1235.0.0 #3388

Merged

rkt: add experimental support for attachable applications #3396

Merged

lucab removed this from the v1.20.0 milestone Nov 23, 2016

euank mentioned this issue Dec 13, 2016

journald strips trailing spaces #3473

Open

lucab closed this as completed in #3396 Feb 2, 2017

jonboulle unassigned lucab Apr 6, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

stage0: attach to the app's stdin/stdout/stderr #1799

stage0: attach to the app's stdin/stdout/stderr #1799

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stage0: attach to the app's stdin/stdout/stderr #1799

stage0: attach to the app's stdin/stdout/stderr #1799

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

User-heavy manual attach

First-class rkt attach

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!