Dante/Developer Journal

Project UDG 2.0

A few months ago I created UDG (Unreal docs generator). I didn't like it much, and the parsing was quite primitive. A few days ago, I found a nice tool which was former developed by sun: javacc (java compiler compiler )

So instead of handwriting the lexer/parser stuff, I'm rather writing the grammar for unrealscript.

Javacc uses a notation which seems to be translatable to EBNF. Unlike some other parser generators, the generated source code also doesn't depend on any libraries besides the standard one.

I hope I will continue this project

Update: Well well, javacc wasn't the right tool for the job. But handwriting was! The parser's up and running.

If you want to comment, place it below this page or below the day you want to comment.

May, 26th 2003

It'll still take a while to get javacc to make all things I want it to do. Absolut posted some ideas of changing DA on the internal forums. He tried to mix my and his interests with the DA concept. The team didn't react very positive about it (I expected that). So I will not lead code there. Some of the team members really had some crazy ideas. Everything they want can be done with, of course, enormous amount of work. They're mappers and obviously got the easiest part in all that work (mapping a space area isn't that complicated I guess). The modellers would have a hell lot of work. The poor coders (not me, I'm part-time will cry rivers.

May, 21th 2003

• Updated UDGs new grammar too: States are not parsed too.

Had a lot of problems with unrealscript allowing "reserved" words as names for variables/methods. Fixed that. The list of recognized tokens is HUGE - I feel sorry for the guy who coded ucc. I tested my grammar on core and engine - it did pass without errors. Next step: generating a parse tree with the help of jjtree (a tool that comes with javacc). Argh - I found DominationPoint_rc which doesn't "\n" at the last exec. Also found a class which didn't have a closing "}" for defaultproperties...

May, 20th 2003

I somehow dislike my math courses. Absolut from [Defence Alliance] and I talked about changing DA a bit, so he wants me to be Lead coder again. I don't know if I want to do that again

DA isn't as interesting to me as it was in UT. Maybe I start a new mod, or join a existing one.

May, 20th 2003

Tested only with "Object", "Actor" and "Playercontroller":

- Parses "class" statement and all those nativereplication, abstract, ... things

- Handles "var" statements with various modifiers like input, transient, ...

- Parses enums, also in combination with var

- Parses structs

- Parses events, functions and operators with parameters, ignores method bodies. I found a "event function" in PlayerController - wtf ? Still - I changed grammar to match them too.

Even thought I've never written such a big grammar (only small one's in one of my courses) I'm quite happy with the current thing.

May, 30th 2003

To have a working Documentation compiler still will take some time. My damn monitor suicided. I used an old 14" thing, which did strange things with light pictures. Today I bought a new one. Really OT: I finally watched Matrix reloaded on tuesday. Except the dancing scene and the "end", the movie is great. I still think it's funny that mankind darkened the sky, althought every form of life is based on it. And it's funny that the machines are using us as energy source, even thought a few power plants will provide more energy in an easier way.

Juni, 2nd 2003

I finished my proseminar work on the subject "HTN Planning". "Bridge Baron" is a Contract Bridge playing program which uses HTN. Hierarchical Task Network Planning is proven to be more expressive than state machines. Maybe an implementation for UT would be cool. It would allow mod developers to add new features to the bots. But since UT hasn't static variables one would need to place those things into levels, or subclass a bot. This would also mean that there could be different bots (codewise) which will still work with a mod without any special work. The basic bot would just have to provide some basic Tasks without Methods that decompose those.

Juni, 29th 2003

There's an error in the text above. I didn't finish my proseminar really yet. Actually I just have to return versions of it over and over again. I decided that the next version of UDG should be open source. Also, it should no longer produce html directly. Instead, it will create xml output which can be postprocessed by xslt code. I actually did some work with my planning bots

I was too much thinking in the turn based way of making decision. That won't work with UT very well as it would have to calculate too many things for making decision. I also found another error above, of course the basic bots would need to have methods to decompose at least the given tasks to primitive operators.

Bah - the DA team finally made me rejoin the team as coder. I'm still not very motivated, and I told them. And I know that people like me, who aren't motivated, don't help a specific mod (like DA in this case) much. Maybe I should take a break from the UT scene like 2cool did.

Oh and my "new" monitor sucks big time. I bought an Hyundai Q910. Actually I called them for warranty 2 times. Both monitors I got had the same failure. At least their support is nice and I don't have to pay even 1 eurocent for anything.

Juli, 26th 2003

Ok finally the proseminar is over. Hell lot of work.

I did alot of coding for DA again. Mainly weapon stuff - the really bad thing in UT2003. UT2003's weapon code is a good example for how it shouldn't be done IMO.

I found out that jeroen (2cool4-u) is now lead coder on Deathball. I think he's hard working and really deserved it. I also guess that he did take that break from modding to get some distance between DA and himself. He wasn't interested in DA anymore, but I guess people from DA (most of them are really nice) kept asking him to help. So he decided to "leave". Clever boy

UDG also received some time. I dropped the idea of writing a parser with javacc. It's a great tool, but it doesn't allow something like inheritable Token constructs. Therefore matching on "event" doesn't imply that it's a <IDENTIFER>. But unrealscript allows those names (example is the string event in class Actor). I therefore decided to write my own Scanner/Parser. The scanner is working and is one big funny list of regexps. The parser is a recursive one and does some weird things.

Juli, 29th 2003

Although I'm going to rewrite the scanner anyway I already had to fix a not documented "feature" in java. Large matchings sometimes throw StackExceptions. Obviously comments are vulnerable for this.

Method parsing is now working well. I'm not using the parsed method modifiers yet. Also method parameters aren't parsed yet - I think I'll implement this today.

Update: I really updated it. Also added delegate. Also states, and methods within them are parsed.

Juli, 31th 2003

I would still call this 30th but it's already 0:20 am. Crazy as I am I rewrote the parser and I'm now using much of the [UnrealScript Grammar] info. That grammar seems to have some minor flaws but my parser seems to work since I only use it as a reference. I already started writing the new parser yesterday. And I made fast progress: It parses consts/structs/enums/states/functions/events/delegates.

August, 5th 2003

Again a exam was on my schedule. Case closed. I had no time to work on UDG. Currently I'm also busy with DA.

August, 19th 2003

Well I worked alot on DA. Before I returned to DA, there were a lot of internal builds (using buildnumbers which weren't any). But all those releases had only small updates, well I don't need to hand out a file to each team member each time I rebuilt. So we were going to make a bigger IR release. DA seems to become some sort of hyper Assault mod. Our mappers want tons of features, but then they should naturally be able to handle them too. Why I'll like trial and error learning it's not recommend when they could ask me instead of producing some strange things in maps.

UDG - well it had a few minor bugs which were enough to stop me coding on it. After downloading and testng UClasses I'm a bit more motivated. Although UClasses seems to be nice program, it takes ages to load the class tree. I guess it should usually show variables etc. but maybe it doesn't work on UT2k3.

After fixing it now seems to detect methods too. Actually a problem in my state parsing consumed necessary tokens.

Ok, I just parsed core, engine and unrealgame and the parser did reach 'defaultroperties' or EOF everytime. Since I don't parse defaultproperties this means it has parsed everything but CppTexts, method bodies or anything like that.

Update: I've rewritten the scanner which used some regexp combo. I knew that it would be slow and it was very slow. The new scanner is 3-5 times faster.

August, 20th 2003

Lol, exactly 1337 classes in my editpackages (wihout my mods). Parsing + name resolving takes about 8 seconds which is pretty fast compared to the old UDG which had a far simpler parser and scanner. Well before yesterday, UDG was just meant to be a documentation generating engine. But after looking at UClasses and UnCodeX and after the mucho of work I invested, I decided to make it a more advanced tool. Just if anybody reads this, I'm still doing it for my own eduction and due to the fact that I'm just a bit weird (Like most coders /me runs). Oh - and because I'm changing the "feature list" of this project, I'm going to rename it too I just need a good name...

August, 25th 2003

I was on a LAN this weekend, so no progress. I was told that UDG is a finalist in the "tools" category of the MSU contest. I'm happy about that. It also motivates me to make a powerful UDG 2. I planned to make a GUI for it since I can actually do things now that weren't possible on the old version. On the other hand, I did some recoding of the parser which I'm going to revert because it's buggy and sucks.

Close before 26th The parser is running fine again, I also have some GUI elements already, mainly package and class tree stuff. I started a html exporter, but I'm unsure about the current implementation. I also want to prevent an information overflow. I also made a short hack and included a class viewer. It doesn't do any highlighting yet since I'm concentrating on the html exporter.

August, 29th 2003

It's all about design decisions. At first I decided to resolve all identifiers after the parsing stage. But I'm currently rearranging the code to dynamically resolve. The new UDG should be able to run while one edits, and only reparse changed files. That would've meant that I'd had to resolve the complete package structure again (the old way). Now I just parse the class and everything should be fine. I didn't yet test the system against "wrong" code, some of the dynamic resolving might crash on it, but I'll see.

September, 5th 2003

Ok the dynamic resolving seems to work ok. I'm using caches to speed resolving up. I also nearly finished my html export function. But it's sloooow. I thought the java DOM implementation was quiet fast. I was disabused, it's the part slowing everything down. Exporting takes 2 minutes. This is kind of funny compared to the fact that parsing takes 5-10 seconds.

Other news: We had a internal release of our mod. Only minor bugs were found, many of them because of not update-to-date UT2k3 installations. Many "suggestions" were posted as if they'd be bugs. It's a bit annoying to filter out the real bugs.

September, 8th 2003

Ok, DOM was slow and my resolving too because I tried to resolve each Token to classes, structs etc. The old resolve-mechanism tried to resolve it to a thing if it didn't know it yet which lead to huge performance loss. The html export is now nearly finished.

I also found a little problem with my resolving. Since 2 packages can have classes with same classnames it's difficult to "remember" them both. Currently resolving uses the most current one which is bad of course. To solve this I will introduce package based resolving which will resolve to the most "current" item at the point the package was parsed.

Following there's a little snapshot from the current version. It shows the package tree view, the class comment tooltip and the source viewer.

September, 11th 2003

Wow, DA is cancelled. Kind of bad since I did a lot of work for it. But I still wasn't very motivated. I'm going to start a new mod with some members of DA. Let's see how it works out.

MythOpus: Aww man. That sucks... good luck with the new mod though

Dante: Thanks, I hope we'll make it

September, 19th 2003

Wow 8 days passed again since the last update. We're finally come up with 2 mods. 1 Big one and one smaller one (which will be more fun playing imo). We'll start with the smaller one and maybe continue the bigger one. No news on UDG. Well that is because I'm already using the html output.

October, 21th 2003

Well a few more days passed I fixed many bugs from UDG. The codebase needs a complete rework... Other than that, I decided to release a preview version of UDG. It's not feature-complete and it's very unstable. It properly won't crash but also won't do what you want. Ok, I'm preparing for release now...

Check http://www.uni-ulm.de/~s_dwaldh/ if you're interested!

October, 30th 2003

We released Unreal Squad yesterday. Thought it's a small mod it makes much fun.

---

Category Journal