I've changed this to be assigned to the VTE team, as that seems to be where the problem lies and why so many VTE based terminals have the same problem.  You can find the code that creates and unlinks the vte tmp files around line 153 in the src/vtestream-file.h file of the latest version.

I don't think this is a bug.

Yes, vte keeps its data on disk in /tmp. If it used memory, the data would end
up on disk anyway, just in swap instead. If you're concerned about this, you
could encrypt /tmp in addition to swap.

Related: bug 631685, bug 635142.

I consider this problem to be a design flaw not a bug. No terminal emulation program should be writing to disk without the user's knowledge. To do so is a violation of the trust that has existed over several decades. In fact, by writing this data to disk, it is circumventing the security of any other program that tries to be diligent about keeping any data it displays off the disk, such as gpg or an encrypted filesystem.

I'm going to be posting a report to bugtraq@securityfocus.com and a few other places today because the public at large needs to know about this issue so that they can clean up any compromised data.  For the sake of future usage of libvte, I'd like to recommend that you revert the changes made to the library that write data to disk. Otherwise this will continue to be a problem.

It seems that the motivation behind the change was to allow for very large or "unlimited" scrollback buffers. Anyone who actually needs such a large scrollback buffer can buy the memory to support it for their corner case. I'm a system administrator and I've only been using a scrollback of 5000 for over 10 years now and have been fine with it. If people really need to go back and look at historical data, there are other commands that can help them with that, such as script, screen, tmux, etc.

What we need to do to address most of the concerns is:

1. Add a buffering layer to vtestream.  Perhaps mmap-based.

2. Add a zlib compressing layer to vtestream.

3. Add a DES3 encryption layer, with runtime generated one-time keys to vtestream.

Doind those should address the concerns in this bug too, without being a performance hit.

BTW, memory fragmentation was another reason this was done.
BTW, your kernel may still write the terminal's memory to your swap partition, so the GPG argument is partially moot anyway.

What needs to be done is you need to remove the code that writes scrollback buffer data to disk at all. If there is a reason other than memory conservation for doing so, then please say so. I would think that people would be more concerned about performance due to disk writes anyways. And some people turn off their swap space to avoid issues like this anyways so your comment about that is moot.

If you an't constructively comment on / criticize my proposed plan, I don't see why I need to reply really.  You are more than welcome to fork vte and let the community at large / distributions decide which version they want to use.

I am constructively criticizing it. I'm saying its a bad thing and that it worked fine before the change. If you're expecting me to only endorse your plan, then you're not being open to the possibility that this simply the wrong way to do it. Many popular terminals work just fine and quite efficiently without writing their scrollback buffers to disk.

There is no reason to complicate the matter further by adding further CPU and disk time to managing the scrollback buffer data through the use of encryption just so that you can put it on the disk and have an "unlimited" buffer. Almost nobody really needs an "unlimited" buffer. It was a bad idea. Time to kill it.

(In reply to comment #7)
> I am constructively criticizing it.

How is "What needs to be done is you need to remove the code" constructive criticism?  Don't tell me what to do.  Tell me why what you're offering (if anything new) is better.

> I'm saying its a bad thing and that it worked fine before the change.

Really?  How do *you* know?  Sure, it worked fine for *you*.  Not for the many other sysadmins, who have tens if not hundreds of tabs, open for over month.  The memory fragmentation caused by vte was swapping their systems to death.  There are many other situations that I tried to address with my design.  But I don't think listing them all is going to make much difference in this conversation.


> If you're expecting me to only endorse your
> plan, then you're not being open to the possibility that this simply the wrong
> way to do it.

Don't tell me what's a wrong solution and what's right.  Tell me what you want!  I already proposed a plan that should address your concerns.  But you simply ignored that.


> Many popular terminals work just fine and quite efficiently
> without writing their scrollback buffers to disk.

You are more than welcome to use those other many popular terminals.  I design vte the way I think it's right for GNOME.


> There is no reason to complicate the matter further by adding further CPU and

It's no more complicated from a user's point of view.  Unless you send patches, I don't see why you should care about the complexity of the implementation.


> disk time to managing the scrollback buffer data through the use of encryption
> just so that you can put it on the disk and have an "unlimited" buffer. Almost
> nobody really needs an "unlimited" buffer. It was a bad idea. Time to kill it.

You just made it clear that you don't know what people want.

> > I'm saying its a bad thing and that it worked fine before the change.
>
> Really?  How do *you* know?  Sure, it worked fine for *you*.  Not for the many
> other sysadmins, who have tens if not hundreds of tabs, open for over month. 
> The memory fragmentation caused by vte was swapping their systems to death. 
> There are many other situations that I tried to address with my design.  But I
> don't think listing them all is going to make much difference in this
> conversation.

You do mention this being a motivation for the change in bug report #631685. But you also later mention that you're not comfortable doing an unlimited scrollback implementation in memory, which seems to indicate what your true motivation is.

I've been a system administrator for 14 years. I currently have 72 bash processes running on my machine, no doubt most of those are terminals all with a 5000 line scrollback buffer. My workstation has been up for 41 days. I have 16GB of RAM (overkill, but I run several VMs) and my swap space is turned off. I know that I can't expect to run this and other X processes within 256MB of RAM. 512MB even. Its not a software problem, its simply a data problem. If people are complaining because their underpowered machine goes into swap when they open up 100s of terminals with 50000 line scrollback buffers utilizing more memory than they have, why should this be the problem of everyone who tries to be diligent about security and has enough RAM for their needs? At some point the user who is making this mistake needs to be told that their expectations are unreasonable.

If you're concerned about angry sysadmins, then I would be more concerned about the even larger number of more angry sysadmins there will be when they find out that their sensitive data is on disk. I know I'm angry about it. I'm trying to be civil, but I also have to be stern so that you'll take me seriously.

I wish I could help you with a patch, however you obviously are a more skilled C coder than I, I'm just trying to push you to do the right thing in regards to the security of data.

(In reply to comment #9)

> You do mention this being a motivation for the change in bug report #631685.
> But you also later mention that you're not comfortable doing an unlimited
> scrollback implementation in memory, which seems to indicate what your true
> motivation is.

Why do you think I have one "true" motivation but mask it with other ones? :)


> I've been a system administrator for 14 years. I currently have 72 bash
> processes running on my machine, no doubt most of those are terminals all with
> a 5000 line scrollback buffer. My workstation has been up for 41 days. I have
> 16GB of RAM (overkill, but I run several VMs) and my swap space is turned off.
> I know that I can't expect to run this and other X processes within 256MB of
> RAM. 512MB even. Its not a software problem, its simply a data problem. If
> people are complaining because their underpowered machine goes into swap when
> they open up 100s of terminals with 50000 line scrollback buffers utilizing
> more memory than they have, why should this be the problem of everyone who
> tries to be diligent about security and has enough RAM for their needs? At some
> point the user who is making this mistake needs to be told that their
> expectations are unreasonable.

You should read what "memory fragmentation" is.



> If you're concerned about angry sysadmins, then I would be more concerned about
> the even larger number of more angry sysadmins there will be when they find out
> that their sensitive data is on disk. I know I'm angry about it. I'm trying to
> be civil, but I also have to be stern so that you'll take me seriously.

You are not adding any value to discussion already existing on bugzilla (in this and other bugs).  I will take you seriously when you respond to my proposed plan, telling me whether it would resolve the issue for you or not.


> I wish I could help you with a patch, however you obviously are a more skilled
> C coder than I, I'm just trying to push you to do the right thing in regards to
> the security of data.

To be honest, I don't even have time for this, but trying to be nice and come up with a simple plan to implement over a weekend or something.  Your "stern"ness or sending more angry sysadmins here wouldn't improve the chances of I working on it.

Note that if you really really prefer keeping your terminal contents in memory, you can just create a tmpfs and direct vte to create its files there; it honours the TMPDIR environment variable (make sure that it's passed to gnome-terminal's factory instance!).

Mark, you're directly responsible for all the time and motivation lost by incoming angry comments on this bug.  I clearly told you that if you comment on my proposed design, I try to find time on a weekend to work on it.  Guess you were more interested in getting publicity instead.

Anyway, let me get this straight for incoming angry people: I am not a regular vte developer.  Have not been for at least two years.  When I picked up vte development / maintenance in 2005ish, it had been unmaintained for years.  These days, there's someone else looking after it at least.  Anyway, there are three features I want to eventually implement on vte, and my proposed design for addressing this bug is one of them.  You can insult me and hope that I fix this, and see how that works out.

Finally, a note re GNOME people and arrogance.  If you never were a Free Software developer, it's easy, expected even, to have a wrong view of how Free Software works.  In short, when you demand that "vte team" shall do something, you are demanding that a bunch of volunteers (1.5 you can say in this case), who have lives BTW, just like you do, to do something because you want it done.  And you claim arrogance when it's not done.  Now, who's being arrogant?  Think about it.

Anyway.  I won't comment further here until the angry mob moves on.  Even then, I probably would just close this bug DUPLICATE and work on the other reports that have much nicer people on them.  People who actually appreciate me volunteering my time to improve software they use.

Thinking about this bug, it feels (to me anyway) like the problem isn't really that VTE uses the disk for scrollback, but that the user didn't expect it to.  People who have been using linux for a while may think they have a pretty good idea how to mop up their private data.  They may know how to turn off ~/.bash_history, how to silence the audit daemon, set up encrypted home directories, and do various other things that keep their private activities private.  These same users may be surprised that there's a new item on their checklist, and agitated that they didn't know about it sooner.  I don't want to talk for Mark, but I could certainly see how that might be true for some users.

I'd propose going the other way, though.  We should store the buffers in ~/.cache and automatically redisplay them when reopening the terminal.  xchat does this for its terminal widget actually, for irc.  Users then will instinctively know their data is getting stored and will know to clear it when necessary or set up luks so it's encrypted or whatever.

We also then have the added feature of not having to run "history" when first opening the terminal to see where we left off.  It's not entirely clear how multiple terminals would work though (unless we bring XSMP into the mix, or something similar, though, bleh)

I also want to add, though, that securing /tmp should be on the standard aforementioned checklist. By its very nature /tmp hosts transient, often security sensitive information.

Behdad, I don't know what to say really regarding you saying that I'm directly responsible for other people coming in here angry about this. Maybe this is where the open source model breaks down. Since people think of themselves as volunteers they don't take as much responsibility on very big projects like they should be and don't realize the trickle down effect you can have on so many other people. I guess I expect that since GNOME is used by literally millions of people around the world that it would mean that there are developers behind the project that take problems like this seriously. gnome-terminal, being the default terminal emulator used on most distributions of Linux is used by millions of people as well.

We all have lives to deal with and I wouldn't expect you to give that up or something. But at the same time, I feel that if someone is going to be allowed to mess with software that has existed for over 10 years, is the de-facto standard used by millions that a little more thought would go into how the software has been used in the past. I guess the concept of terminal data never hitting the disk skipped a generation or something. For every person who agrees that this is a huge problem, there are 5 others that don't understand why this is a problem and think its just a matter of root seeing the files. They don't realize that so many programs and paradigms of computing are based on the data in the terminal only being in RAM. Maybe that was wrong of all of us to assume that, maybe we should have written this rule down somewhere. But nevertheless its the way things have been for a very long time, which is why many of us think that you are doing it wrong.

Right now, as we speak, there are millions of hard drives out there with data that was never intended to be on them due to this problem. I've already handled my own breaches and don't plan on using any libvte terminals.  So what I'm really trying to get out of this is security for other people in the future and also because everyone who is affected deserves to know that it was a problem, whether they plan on doing anything about it or not. There are definitely people out there freaked out right now about this.

Ray, probably the user didn't expect it to write the scrollback to disk because with the exception of Konsole and from what I've heard, Apple's Terminal.app, none did. So its a reasonable expectation for the user to have, don't you think?

Disclaimer: I'm making that statement that none did without actually researching it. But upon testing several for this report, Konsole and libvte based ones where the only ones.

If you want to make libvte continue writing to disk, do what you want. But users should be aware of that "feature" and the direction of libvte.

Created attachment 209285 [details] [review]
Use memory-backed stream for non-infinite case

Here is an insane patch that fixes the problem. I just did a behavioral replacement of file operations with memory operations. This patch could be seriously improved if I could understand the interaction of "offset" and "newpage" better, and just make this a sensible moving window of memory instead. I ran out of time last night, though. Delaying the stream creation was the trick, though.

Patch header reads:

use memory-based circular buffer "stream" for the non-infinite
scrollback state. This delays stream initialization so that the true size
of the desired scrollback state can be discovered. This also means that
changing the scrollback size in the middle of using a stream means you
won't change the backend stream type.

I guess my opinion on whether or not it's a reasonable expectation for users to think that terminal buffers are fully transient, doesn't really matter.  Some users clearly have that expectation since this bug report exists, after all.  

I will hazard to say that most users don't care either way, though.  Certainly, I don't think anyone who has a .bash_history, audit log, unencrypted swap, or whatever should care.  

Furthermore, I think those people who do care about this should also have concern about /tmp more generally.  setuid programs and other security sensitive programs use mkstemp() etc all the time.  People who care should have /tmp mounted with tmpfs, should have encrypted swap, should use pam_namespace to make /tmp per user, and other security hardening things like that.  If they're doing that, this problem gets covered "for free"

So to reiterate, my opinion is, we could fix this type of misunderstanding by keeping the buffer persistent across restarts, and we probably should.  I don't see this as a super high priority issue, though.

I think Behdad’s initial idea with encryption was the best one. I hope you’re not put off by all this self-righteous posing. Ray’s idea is also interesting but I’m not in favor.

Hope that nailing search is among the three things mentioned for the future, Behdad. Thanks.

(In reply to comment #23)
> Created an attachment (id=209285) [details] [review]
> Use memory-backed stream for non-infinite case
> 
> Here is an insane patch that fixes the problem. I just did a behavioral
> replacement of file operations with memory operations. This patch could be
> seriously improved if I could understand the interaction of "offset" and
> "newpage" better, and just make this a sensible moving window of memory
> instead. I ran out of time last night, though. Delaying the stream creation was
> the trick, though.
> 
> Patch header reads:
> 
> use memory-based circular buffer "stream" for the non-infinite
> scrollback state. This delays stream initialization so that the true size
> of the desired scrollback state can be discovered. This also means that
> changing the scrollback size in the middle of using a stream means you
> won't change the backend stream type.

Thanks Kees.  I'll take a look it this weekend to see what I can do.  Basically the 'oofset' is a growing index into where the last byte written to the stream is logically.  Each page has a start offset.  Your patch looks correct from a quick look, but as you said, for memory streams it can be simplified considerably.  Another option is to use fmemopen kind of stuff, but I have not seriously thought about it yet.

Malloc failure is a concern though.  The nice thing about the file implementation is that if you run out of disk spae, nothing breaks, you just get empty areas in your scrollback buffer.  Crashing the process in case of OOM is definitely not an option.

But as I said before, the design I like to see eventually is adding caching, compression, and encryption layers to the stream implementation.

There's enough demand for the "unlimited scrollback" idea that the spilling to disk implementation was done in the first place.  The systems I use heavy scrollback on and care about security have encrypted /tmp, swap, and everything else.  Ray Strode's comments in #24 are not a workaround hack; they are what any systems administrator with serious security concerns should have *already* been doing here--and should continue to do even if this particular issue is resolved.  The only one that's not common knowledge is that it's important to worry about isolating tmp on multi-user systems, and how to do that on Linux.  Here are some examples of the incantations needed to set that up:  http://www.ibm.com/developerworks/linux/library/l-polyinstantiation/ http://fedoraproject.org/wiki/Infrastructure/FedoraPeopleConfig http://etbe.coker.com.au/2008/12/13/per-process-namespaces-pam-namespace/
  Christian Persch's suggestion of setting TMPDIR is an easier way to resolve the /tmp sharing problem for the vte case, but you're likely to be exposed to it via other software too.

The idea Mark Krenz advocated, that only a memory based implementation is acceptable here, seems bizarre to me.  Why should people who are using this feature happily and securely lose it because others didn't think "hey, I wonder where the potentially 'infinite scrollback' is going" until now?  I wish people would take some ownership for the security of their systems already; this is not some some obscure hidden feature.  If you do important things in your terminal and didn't think about where your scrollback was being stored at until recently, consider this a lesson in what due diligence looks like.  You shouldn't be pointing fingers; you should be asking yourself "I wonder what other things are being written to /tmp I never considered before?"  The page Mark put up popularizing this issue suggestions running "strings" on your whole tmp partition.  Bet you'll more than just your scrollback data worrysome there.

Behdad's originally proposed solution seems intent on reinventing disk encryption.  That has some benefit from the perspective of the library solving this problem for all of its users, as well as resulting in something that could be released as a backport to older systems.  I'd be content with making it more obvious where the scrollback was going to and making sure it was easy to relocate for people, such as the idea Ray suggested for putting it into ~.cache by default.  Even though vte can't enforce that UI being exposed, GNOME's terminal could lead by example there:  have a clearly visible section in the scrollback setup section stating where it's being stored.  I recognize that idea would make for a difficult backport though, having already been exposed here before on things like how dialog box text is frozen for translation.

So, for what it's worth, I think it's important there be a way to have a strictly memory-based backscroll option available. Even with compression, encryption, caching, etc, it'll still touch disk. I'd like to like have my terminal's normal operation not tied to disk in anyway. Given my harddrives, if there's a way to hit disk contention, I'll find it. ;)

Opening bug up again after removing all non-relevant comments (me, troll, andre).

(In reply to comment #24)
> People who care should have
> /tmp mounted with tmpfs, should have encrypted swap

Well that doesn't help anymore, apparently vte will be using /var/tmp in the future. And even if you encrypt your /var/tmp partition, that's still not equivalent to not storing scrollback on disk -- because /var/tmp needs to persist for the whole uptime of the machine, whereas the scrollback buffer should not live any longer than the terminal window.

(In reply to comment #4)
> 3. Add a DES3 encryption layer, with runtime generated one-time keys to
> vtestream.

Just a small nitpick, please don't use the obsolete DES or 3DES algorithms. They're slow in software and they're no stronger than AES at any key length.
Run 'openssl speed aes des' for a rough benchmark ("des ede3" stands for 3DES):

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
des cbc          69093.02k    72191.96k    72441.69k    72727.89k    72736.77k
des ede3         27737.98k    28176.53k    28322.56k    28370.26k    28485.69k
aes-128 cbc      94046.20k   100687.83k   103012.78k   263552.00k   267059.20k
aes-192 cbc      79652.73k    84045.33k    85701.55k   221515.09k   223335.77k
aes-256 cbc      67496.58k    70595.90k    71769.77k   185105.07k   187574.88k

(In reply to comment #40)
> (In reply to comment #4)
> > 3. Add a DES3 encryption layer, with runtime generated one-time keys to
> > vtestream.
> 
> Just a small nitpick, please don't use the obsolete DES or 3DES algorithms.
> They're slow in software and they're no stronger than AES at any key length.
> Run 'openssl speed aes des' for a rough benchmark ("des ede3" stands for 3DES):
> 
> type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
> des cbc          69093.02k    72191.96k    72441.69k    72727.89k    72736.77k
> des ede3         27737.98k    28176.53k    28322.56k    28370.26k    28485.69k
> aes-128 cbc      94046.20k   100687.83k   103012.78k   263552.00k   267059.20k
> aes-192 cbc      79652.73k    84045.33k    85701.55k   221515.09k   223335.77k
> aes-256 cbc      67496.58k    70595.90k    71769.77k   185105.07k   187574.88k

Thanks.  Good to know.  Maybe I hack on this on the plane today...

My opinion on the encryption solution is that no program in the terminal will ever be more secure than how secure libvte is. This is essentially the way it has always been with terminals. No matter what algorithm you choose, no program's efforts to be more secure will matter much. If you're going to encrypt the data, you have to keep up with the encryption algorithms. Eventually, even AES-256 will be crackable in by a consumer grade computer, perhaps by quantum computers that people are already talking about. If you think this sounds far fetched, consider that only 15 years ago DES was still not considered crackable.
 
The other issue is that the data on disk becomes unmaintained once you delete the inode for it. As soon as libvte lets go, there is no way to go back and re-encrypt the data with better algorithms.  So you could have data lying on a drive for a long time that was encrypted using a cipher that is later considered weak.

I agree that encryption can solve many problems and is generally a good thing, but this is a different situation than most use cases.

I came here trying to figure out why the computer next to my bed is writing to disk every few seconds while I am trying to sleep.  I was very surprised to discover that gnome-terminal was the culprit.  In my opinion, the "security" aspect of this bug report is a red herring.

Greg Smith brings up a good point in #32 that the writing to disk is done in order to support unlimited scrollback.  The default 512 line scrollback seems like a small enough number of lines to keep in memory.  I don't want or need unlimited scrollback, so it seems silly to pay for it if I am not using it.

In the future I will be using tmpfs on /tmp and /var/tmp to avoid this problem.

What is the status of Kees Cook's proposed patch in #23?  Is it in conflict with the unlimited scrollback feature?  If not, I would very much like to see it applied.

See comment 4 for how it's planned to address this.

See bug 738601 comment 28 - it seems like a totally irrelevant refactoring will make it very easy to add encryption.

Encryption of the tmp file is implemented now in bug 738601, expected to land in vte-0.39.1.

The scrollback files' contents are encrypted now (using AES 256 GCM), will be released in vte-0.39.2.  So marking this bug as fixed.

The other side issue mentioned here (constant disk access) is tracked in bug 631685.