Log format and viewer rewrite

Probably should go along with the log format rewrite.

Would be nice if the log viewer could group the chat logs according to meta grouping in the buddy list, and to display their alias rather than their user name in the "contacts" list.

We need to fix logging, basically. Message history won't work right once we kill sql logger, too.

See LogFormatIdeas for discussion.

Setting to 'task', which I'm gonna use to mark 1.0 projects.

Ticket #400 (http://trac.adiumx.com/ticket/400) should be kept in mind when rewriting the log module. I opened it as change request for the current logging and it was closed as irrelevant for the moment being. The ticked focused on keeping the history in UTF-8 as a standard file encoding to be human readable.

first off, no need to provide a link to the ticket, it's automatically linked. trac++ This isn't necessarily a bad idea, but I'm not sure of the correctness of leaving unescaped characters in XML files. I'll investigate it.

I would like to suggest that the Adium logs be moved to the standard place for log files on OS X.

Most applications store their log files in ~/Library/Logs/

See #1702 for an explanation of why we will not be storing our chat logs in ~/Library/Logs

Condensing the other logging tickets to here, since it's all related to the rewrite.

Request for the new log format. I recently discovered that you are about to switch to a new log format. I am quite happy with the current format but I see our reasons for the change. I will no doubt go along for the ride. However, I am very fond of my log and would hate to lose it. therefore I request (actually, I kindly ask for it) an importer of some kind. Or at least support for the old html format.

Thanks!

There will definitely be an importer for the old log format. It will probably be done as part of the first run wizard (#9). This is also mentioned in the description.

I've posted on this in the forums, but I'm concerned about this new format.

I think it should incorporate the following information:

- It should log all timestamps and do so in GMT with timezone offset. Traveling cross country has tainted my logs. - It should retain all original formatting, colors, etc - it should retain the sender and recipient's screen name AND full name

XML-based would be nice. Ideally it would be a format that is easy to parse and use from the command line. I often find myself searching logs in the terminal using grep because Adium's Log Viewer is broken. The HTML log format thus far has been a pain but at least it is somewhat usable from the command line.

Updates:

1) This has been addressed, see [XMLLogFormat].

2) IMO, if you're having to resort to grep to search your logs, that's a defect in Adium :) I"m not particularly concerned with making our new logs greppable, and would instead like to focus on making Adium's log viewer rock :)

that's XMLLogFormat, sorry.

would be a nice thing too have a working message history in 1.0svn, as it would result in "better betatesters". they could use 1.0svn for "everyday use" then.

What I like about the logs today is that you can open them in your web browser and read them. This does not seem to be possible with this new format. One way to accomplish this is to link each logfile to an XSLT file provided with Adium.

What extension will you be using btw? I guess double clicking won't work like today, but you could always drag'n'drop the lof file to the browser.

anonymous: 1.0svn is not for use by the general public. beta-testing is only done with betas.

XSLT isn't necessary. The new log files as of the current spec would retain the ability to be read by modern web browsers. CSS alone would work nicely in Safari. Regardless, the new log viewer will hopefully remove any need to use Safari for reading logs.

changing the mock-up URL to a link.

fie. Trac still was trying to show that as an image.

Unassigning, since cbarrett is taking a break.

Comment: Should we store group chats separate somehow? right now it's hard to tell that "durintest" was a group chat room on AIM. Plus, what happens when you join the durin42 group chat room on AIM? Right now, I think the answer is armageddon...

I guess since I actively worked on this in the last day, I"ll re-assign... just like me to be wishy washy :P

Group chats are gonna be stored separately. I'll work on getting the spec at least partially prepared for that.

Should the new log viewer have a calendar control?

(In [15594]) 60% or so of an improved log viewer with iTunes-browser style tables at the top. I'm committing it now mostly to get feedback on its direction and the improvements thus far.

I rejected the Mail.app-search-style top type (from, to, content, etc.) bar because it felt too restricting, but I'm open to suggestions.

Note that the (Date) column is not implemented. Rather than listing every possible date, I picture it having "Today", "Yesterday", "This Week", "This Month", and so on.

In the process, much of the old log viewer moved to AIAbstractLogViewerWindowController, which is really only abstract in the sense that it is a common base class for the two log viewers (10.3 and 10.4) we now support, appropriate since the bottom half of the new one looks like and works like the old one.

Along with this commit we also see a huge speed-up in loading of logs in the old-fashioned, non-searchkit way (which is still used, in a modified form, for an initial 'search' of no criteria so is relevant on 10.4).

Refs #38.

Cool progress and all, but the new UI feels *really* big. It seems like having <50% of the available space devoted to the primary purpose (reading logs) isn't such a great idea.

Yeah, I can see that. Would it help if the Browser area were togglable, like the iTunes Browser on which the UI is modeled?

Yes, that would help. The other reason why the iTunes one is acceptable (I'm not a huge fan of it there either, but it works) is because it's one row of stuff instead of two. Pulling the split view up so the top pane is closed makes the new log viewer a lot cleaner looking.

How about displaying the log text in the selected message theme?

• The log viewer needs to be fast for displaying a whole conversation; processing an entire previous conversation in the webkit message style could be slow
• I personally use the log viewer for copy/paste in many cases because our current webkit copy/paste is pretty bad for spacing and such... we'd need to have that fixed.
• In general, I feel like a simple view for the logs is better than the more complex current-conversation appearance, but that could just be me.

Ick, that was me in the previous comment.

copy/paste isn't always that great from the log viewer. The case that comes up for me a lot is the Sametime/meanwhile case where in our case the user names are huge:

11:05:26 uid=foo, ou=active, ou=employees, ou=people, o=example.com: yes

That's a lot of ID for not very much content. If there was some way to see the alias rather than the user name in the log viewer it would be nice (slight extension of the request far above for having the alias in the contact selector).

Is there any way we could set it up so one can keep an encrypted copy of OTR chat sessions in the new logging system?

I personally prefer that all of my chats be logged as plain text, so the currently proposed fix of disabling the logging of encrypted chats by default, while allowing one to re-enable logging, works for me. However, I can easily imagine wanting to keep a conversation for the future, but not wanting it to be stored as plain-text locally.

(In [16007])

• Reverted most of [15407], [15475], [15594], and [15602]. Lots of work for a really bad result. Some of the old code is still lying around unused because I'm going to be moving it around and using it in an improved log viewer soon.
• Improved speed of the old log viewer
• Fixed our index type to be kSKIndexInverted instead of kSKIndexVector. Inverted indexes are for mapping terms to documents, which we want; vector indexes are for similarity searches, mapping documents to terms. Inverted indexes are faster for term-based searches (which are the only kind we do) and also support boolean and phrase searching in addition to Google-style search queries in 10.4 and up.
• We now use the Spotlight importer when indexing content. In 10.3, our logs will be read as plaintext by the default SearchKit importer, which is fine since that's the behavior we were manually implementing in the first place. In 10.4 and up, markup will be removed before indexing, and future improvements to our Spotlight importer -- such as supporting parsing of the XML format -- will be automatically utilized.

Refs #38.

(In [16009]) AIMDLogViewerWindowController (whose name is now improper but will do for the time being) now uses the 10.4+ SearchKit methods for fast asynchronous searching. Most of AILogViewerWindowController has been moved (with very minimal modifications) to AIAbstractLogViewerWindowController. The only difference between the two subclasses is now (and hopefully will remain) the precise mechanism by which searching is accomplished.

Check out the insane speed on pure-content searches in 10.4! Summary of speed boosts: proper index style, 10.4+ method usage, improved laziness of AIChatLog. Next up, filtering by one or more contacts, metacontacts, or date ranges, probably with a UI overhaul of the log viewer.

Refs #38.

(In [16016]) Most of a shiny new log viewer UI.

• Note the support for speedy filtering for multiple-selection of contacts, including while doing content searches :)
• Once you've signed on, opening the log viewer will show contacts organized by metacontact, an organization upon which you can search, too.

Refs #38

Much much better. Searching still seems to be missing some content for me though. I did a search by content for "the" and got 9 results...

Struck out completed portions... that's the big ones related to the log viewer :)

- I have a small feature request about the new log viewer : when I click on a contact name, i'd like to be able to see the whole IM history, in case of just one (the selected day) or 0 (if i unselect it)

so can you allow multiple selection or IM sessions ?

it can be useful for printing, but mostly it's more convenient to read and send large logs to somebody (i did it yesterday ... i had to copy and paste about 50 days of IM in a word document ... so not fun ...)

- a small bug in current svn : when i type a few words in the search box (eg : "hello honey") then search will work pretty well, but highlight separate words ... (eg : only hello, or honey, or even none of them, but the problem is that it can highlight one of these two words when they are in separate sentences of the same day log !!! (you can be sure that that log contains the "hello honey" sentence somewhere tho)

i hope you can fix that

thanks :)

I don't understand what "when I click on a contact name, i'd like to be able to see the whole IM history, in case of just one (the selected day) or 0 (if i unselect it)" means.

Sorry Evan, let me explai it more precisely : When I click on a contact name in the new log viewer, the current behaviour is that it automatically displays the latest chat session with that contact. That's fine like that But i would love to be able to select a lot of different chat sessions at the same time (each chat session will have to be separated by the date of the day, or to display date and hour before hach line in case of just hours.

I'm asking that because now i feel quite limited in the log viewer, as explained before, sometimes it'll be cool to be able to copy and paste, or to print a lot of chat sessiosn at the same time.

I hope you see what i mean now :)

I see -- you're requesting a new feature, not reporting a bug in the new log viewer, since this part of the display behavior is wholly unchanged from the old log viewer.

Yes, sorry :)

(In [16224])

• Multiple logs can now be selected and displayed; date/contact information is shown above each displayed log when multiple logs are selected.
• Quoted search terms now highlight properly when doing content searches
• Various performance and buglet fixes

[16224] also adds printing support to the log viewer.

WOW thanks a lot that's just amazing :)

one last thing (i promise :-)) : can you make the contact column larger ?

Thanks again :)

selecting a large number of logs cause adium to crash, everytime (i selected 120 logs with select-all, but actually it crashed too with much less logs too)

i can attach a crash log if you need it (adium crash reporter doesn't launch, but apple crash reporter does)

Replying to adium:

selecting a large number of logs cause adium to crash, everytime (i selected 120 logs with select-all, but actually it crashed too with much less logs too) i can attach a crash log if you need it (adium crash reporter doesn't launch, but apple crash reporter does)

Is this with svn head or with .89.1 or something else?

Crashlog would be nice. I can reproduce this, but it's weird as heck. I think it has to do with the log viewer using huuuuuge amount of ram, which I *thought* I fixed with the autorelease pool I added, but apparently not.

"No memory available to program: call to vm_allocate failed" <-- eep!

here is it, I hope it will help you

you are right, Adium crashed after using up to 120mb or ram (from 16mb usually) ;)

i'm using SVN (compiled 16/06/06)

http://files.bultez.info/adium/Adium.real.crash.log

Replying to sam:

http://files.bultez.info/adium/Adium.real.crash.log

The bottom crash log in that file is the one we're interested in. I was confused until I realized there were multiple crash logs in that file.

Replying to cbarrett:

Replying to sam:

http://files.bultez.info/adium/Adium.real.crash.log

The bottom crash log in that file is the one we're interested in. I was confused until I realized there were multiple crash logs in that file.

He has edited it.

Replying to sam:

you are right, Adium crashed after using up to 120mb or ram (from 16mb usually) ;)

Indeed. That definitely looks like an out-of-memory problem.

Please type this into a Terminal window when Adium begins to eat RAM: heap Adium > heap_adium.txt

It will take a minute or two to do its work (maybe more in the problem case). When it finishes, upload that file (heap_adium.txt) to your website, and comment here with the URL.

i just compiled Adium again. (for the record, i just get the code with svn, and then type make in /release)

i selected all my logs (1100), saw the ram usage starting to increase, launched boredzo's command (it was done is a few seconds) then adium ate up to 95mb of ram, but didn't crash !! ;) and the log viewer successfully displayed all the selected logs

the problem still exists, because adium is still using that amount of ram now

anyway, i hope this file will help you :) http://files.bultez.info/adium/heap_adium.txt

Replying to adium:

anyway, i hope this file will help you :) http://files.bultez.info/adium/heap_adium.txt

Unless the problem lies in the non-object allocations, that file isn't saying much that's surprising. You have a lot of logs selected, and there are a lot of logs (and log-related objects) in memory.

I did find it interesting that there were 1750 AIChatLog objects, though.

it crashed as soon as i quit adium

i tried that a few times more, and sometimes it works, sometimes it crash, sometimes mem usage stay low, sometimes it rises and rises ....

so it's kinda random ...

i'm sorry not being able to help you more, ... i never written a single line of code ...

to boredzo : you're saying that i have a lot of logs in memory, ... do you want me to logout, or restart ?

Replying to sam:

i'm sorry not being able to help you more, ... i never written a single line of code ...

You're helping just fine. We can write the code; we just need to figure out the problem first.

Replying to sam:

to boredzo : you're saying that i have a lot of logs in memory, ... do you want me to logout, or restart ?

No. Quitting Adium is enough to free that memory. We should also free it when logs are deselected; I don't know whether we're currently doing that.

i have the same behaviour here, but adium does not crash. it just hangs (spinning pizza). anything i can do to help?

I just checked out the current svn. with the current svn it crashed instead of spinning pizza.

Replying to adium:

i just compiled Adium again. (for the record, i just get the code with svn, and then type make in /release) i selected all my logs (1100), saw the ram usage starting to increase, launched boredzo's command (it was done is a few seconds) then adium ate up to 95mb of ram, but didn't crash !! ;) and the log viewer successfully displayed all the selected logs the problem still exists, because adium is still using that amount of ram now anyway, i hope this file will help you :) http://files.bultez.info/adium/heap_adium.txt

hm.... non-object, eh? Sounds like searchkit to me. I wonder if we've got a loop where we're building up searchkit temporary objects (like we were building up temporary html decoder objects earlier), or if it just takes a lot of ram. If the latter... I wonder how we can cap it/avoid it. I'll take a look at the code and see if I can figure out what's going on; I've been focusing on it from a profiling/objectalloc POV so far, and that's been of mixed usefulness.

Well, MallocDebug turns up a bunch of errors... [[[ ... libMallocDebug[Adium-2971]: frame pointer goes from bffee8a0 to bfffe8f0 -- assuming invalid. libMallocDebug[Adium-2971]: frame pointer goes from bffee8a0 to bfffe8f0 -- assuming invalid. libMallocDebug[Adium-2961]: target application trashed bytes after end of malloc buffer; buffer is at 0x18e7dd0, and is of size 16 libMallocDebug[Adium-2961]: Stack start is f009a000 libMallocDebug[Adium-2961]: Stack size is 00080000 libMallocDebug[Adium-2961]: target application trashed bytes after end of malloc buffer; buffer is at 0x5d6ead0, and is of size 335 libMallocDebug[Adium-2961]: target application recently wrote to freed malloc'd memory at: 0x5d6ec40, suggesting it didn't realize the memory had been freed libMallocDebug[Adium-2961]: target application recently wrote to freed malloc'd memory at: 0x5d6ec40, suggesting it didn't realize the memory had been freed libMallocDebug[Adium-2961]: target application recently wrote to freed malloc'd memory at: 0x5d6ec40, suggesting it didn't realize the memory had been freed libMallocDebug[Adium-2961]: target application recently wrote to freed malloc'd memory at: 0x5d6ec40, suggesting it didn't realize the memory had been freed libMallocDebug[Adium-2961]: target application recently wrote to freed malloc'd memory at: 0x5d6ec40, suggesting it didn't realize the memory had been freed libMallocDebug[Adium-2961]: target application recently wrote to freed malloc'd memory at: 0x5d6ec40, suggesting it didn't realize the memory had been freed libMallocDebug[Adium-2961]: target application recently wrote to freed malloc'd memory at: 0x5d6ec40, suggesting it didn't realize the memory had been freed ... ]]]

Dammit. Using the right formatting chars this time:

Well, MallocDebug? turns up a bunch of errors... ... libMallocDebug[Adium-2971]: frame pointer goes from bffee8a0 to bfffe8f0 -- assuming invalid. libMallocDebug[Adium-2971]: frame pointer goes from bffee8a0 to bfffe8f0 -- assuming invalid. libMallocDebug[Adium-2961]: target application trashed bytes after end of malloc buffer; buffer is at 0x18e7dd0, and is of size 16 libMallocDebug[Adium-2961]: Stack start is f009a000 libMallocDebug[Adium-2961]: Stack size is 00080000 libMallocDebug[Adium-2961]: target application trashed bytes after end of malloc buffer; buffer is at 0x5d6ead0, and is of size 335 libMallocDebug[Adium-2961]: target application recently wrote to freed malloc'd memory at: 0x5d6ec40, suggesting it didn't realize the memory had been freed libMallocDebug[Adium-2961]: target application recently wrote to freed malloc'd memory at: 0x5d6ec40, suggesting it didn't realize the memory had been freed libMallocDebug[Adium-2961]: target application recently wrote to freed malloc'd memory at: 0x5d6ec40, suggesting it didn't realize the memory had been freed libMallocDebug[Adium-2961]: target application recently wrote to freed malloc'd memory at: 0x5d6ec40, suggesting it didn't realize the memory had been freed libMallocDebug[Adium-2961]: target application recently wrote to freed malloc'd memory at: 0x5d6ec40, suggesting it didn't realize the memory had been freed libMallocDebug[Adium-2961]: target application recently wrote to freed malloc'd memory at: 0x5d6ec40, suggesting it didn't realize the memory had been freed libMallocDebug[Adium-2961]: target application recently wrote to freed malloc'd memory at: 0x5d6ec40, suggesting it didn't realize the memory had been freed ...

Bah, that still doesn't look right... :/

ok, it's definitely not running out of heap space or anything like that. I just selected one, then, two, then three, and so on until it crashed... it was around 10-15, which is not enough to cause horrendous memory usage. The stack trace showed it crashing in NSView's visibleRect method, which is BS. What's actually going on is a memory smasher, I think.

(In [16267]) Fixed a memory stomper in -[NSTableView(AITableViewAdditions) arrayOfSelectedItemsUsingSourceArray:] and -[NSOutlineView(AIOutlineViewAdditions) arrayOfSelectedItems]. The latter has been around a long time and is probably responsible for more than a couple unexplained crashes in 0.8x and 1.0svn. The former is responsible for crashes when selecting multiple logs described in #38. Refs #38.

When deleting all logs belonging to a certain contact adium should delete the contacts name in the left column of the log viewer.

you fixed it, thanks :-) I love it now :)

as the new log viewer seems to work pretty well now, let me add some ideas for a future Adium version (after 1.0). Feel free to use some, of not at all ;)

Can you make it look like all the new apps like iTunes, Mail2, NetNewWire ? I find them pretty sleek :)

http://files.bultez.info/adium/nnw.jpg

* the left column, containing contacts, in a light blue color

* the separator line between contacts and the other two parts of the viewer, should be 1px large, black, and allow to resize when dragged

* maybe move the '# or logs' bar from the top to the bottom, and make it look like NNW.

* in that bar, you should display '# of logs (# selected)' and maybe 'from 12 february 2005 to 22 march 2006'

* make sure that each and every contact has an icon (as of today, Gtalk contacts don't have any icon, but it's not specific to log viewer, so i guess it's a non issue)

* the contact column should be larger, and resizing it should redraw instantly the contacts (as of now, it redraws it when we stop resizing it)

* contacts with 0 logs should not be in the contact list, or at least should be deletable

* i prefered when email adress was first, then contact name (it was easier to read)

* i don't think the 'all' contact is useful, because users fill quicktly figure out that we can now select multiple contact and logs

* oh, i just noticed that the 'this week' filter make the week start on sunday. I think you can get that setting from international preferences (in france we start the week on monday)

* selecting a large number of logs still takes time, so it could be great to have a small progression bar on the bottom line

thanks, Sam

Those are some great ideas, Sam. This ticket has gotten pretty unmanageably large -- tickets are much more useful when they are focused. Could you please submit each of your thoughts as a separate ticket so they can be worked on (and marked as completed) in a piecewise fashion?

I'd like for future discussion of the log viewer to be in a ticket or tickets which isn't this one. The Log Viewer portion of this ticket is now complete. Thanks! :)

Replying to evands:

Those are some great ideas, Sam. This ticket has gotten pretty unmanageably large -- tickets are much more useful when they are focused. Could you please submit each of your thoughts as a separate ticket so they can be worked on (and marked as completed) in a piecewise fashion? I'd like for future discussion of the log viewer to be in a ticket or tickets which isn't this one. The Log Viewer portion of this ticket is now complete. Thanks! :)

ok that's done :) feel free to modify milestones of course :) i put UI changes to 1.1 and more background stuff for 1.0

I'm going to mark this ticket closed and split it off into sane parts; a single ticket for the two very different tasks of the new log format and the new log viewer makes no sense.

(In [17535]) Message history now uses the new log format. Still has some minor kinks to be worked out, but works for the most part. I'm leaving the debug logs in for anyone who wants to try and debug it. They should probably be removed before a release. Refs #38. Fixes #4295. Also, in the process, I fixed #4412, so blank logs are no longer created. Spiffy, huh?

A reminder for the tickert #522, which wasn't done even if It could interest some of us.

Indeed, some like me love to sort by log size so people which you talk often are in top of contact list.

I would prefer to show it as a built-in feature instead of usin' plugins.