Delicious flat che^H^H^Hfiles (30)

1 Name: #!/usr/bin/anonymous : 2006-12-02 10:33 ID:SeI56d3E

I was wondering, does anyone here have anything interesting to say about flat files for data storage ? Recommended reading, tips, tricks and advanced concepts...

You see, I'm not a big fan of relational databases. I do understand them very well, actually I pay my bills by working on a quite big database application, it's just that I like the simplicity and beauty of flat files. And real men don't need DBMS anyway. OK, let's not talk about RDBMS.

Say, which markup language / data serialization format do you like ? Have you used xattrs in any project ?

2 Name: #!/usr/bin/anonymous : 2006-12-02 12:16 ID:/lRkt3Yv

Flat files are cool for data which is written out once and then read in again. 3D meshes, configuration files, high score tables, user preference data and so forth.

For things you'd like to modify a record at a time, where reading the file in and writing it out again slightly differently isn't acceptable, there's either something like GDBM or Berkeley DB, or the more heavy and serious RDBMS of your choice. I mean, it's not like most of us would like to spend a year or more putting together a storage engine when people, smart people, have written (and debugged!) such things for us. Plus, Berkeley DB has concurrency control without a server. How cool is that?

3 Name: #!/usr/bin/anonymous : 2006-12-02 15:40 ID:l0kpOZof

I like to write XML files and store them in RDMS

4 Name: #!/usr/bin/anonymous : 2006-12-02 16:38 ID:lPdrDISa

JSON is a cool serialization format. (YAML is ok too but JSON is simpler and good enough)

5 Name: #!/usr/bin/anonymous : 2006-12-02 22:50 ID:O+ba3X+r

>>3
Very good ! I just hope you use Java, otherwise, you know... it wont scale and wont be enterprise leading, paradigm shifting solution.

>>4
Word, JSON is beautiful.

6 Name: #!/usr/bin/anonymous : 2006-12-02 23:05 ID:O+ba3X+r

Did you know that this board stores everything in flat files ?

I didn't have enough time to explore the source, but I believe they go into res[1] directory. It's weird... could it be that Kareha stores posts in XHTML ? Parsing through that shit every time... Also notice how meta data about thread is stored, view source and look at the first line.

[1] http://4-ch.net/code/res/

7 Name: #!/usr/bin/anonymous : 2006-12-03 12:31 ID:Heaven

>>6

Very little is parsed - the full html is stored on a single line, and can just be copied over to the front page. The only thing that is parsed is a regexp which replaces the body text with an abbreviated version when needed.

8 Name: #!/usr/bin/anonymous : 2006-12-04 08:46 ID:SeI56d3E

>>7
I see, that makes sense... still, no matter how you look at it, mixing data with the view seems to be a bad idea.

What if you want to change the view ( beyond CSS ), what if you need more than one view ( like RSS ), where will you store that data which shouldn't be on the view ( for example IP addresses ) ?

9 Name: #!/usr/bin/anonymous : 2006-12-04 12:58 ID:Heaven

>>8

All of that is hard, but: None of that is really necessary.

Hypothesis: Proper modularization and abstraction actually causes feature bloat, by making new features too easy to add.

10 Name: #!/usr/bin/anonymous : 2006-12-04 14:45 ID:KyqYGJQS

Counterpoint: modularization only appears to lead to code bloat because adding features "because they might be needed in the future, although I know of no plan to do so" is so seductive in comparison to doing actual work.

11 Name: #!/usr/bin/anonymous : 2006-12-04 16:21 ID:oUxukLrR

Feature bloat: Who cares? If a feature sucks and nobody uses it, just take the damn thing out again. Life's too short to agonize over these things.

12 Name: #!/usr/bin/anonymous : 2006-12-04 22:06 ID:Heaven

>>10

I'm not sure how that means it only appears to lead to feature bloat. You seem to be agreeing with what I said.

>>11

That's not how feature bloat works.

13 Name: #!/usr/bin/anonymous : 2006-12-05 02:42 ID:oUxukLrR

>>12
Do you mean to say that's not how you work, i.e., you can't bear to remove a useless feature, thinking it might be needed in the future?

If so, use version control. That way the old code is always in svn or darcs or whatever if you change your mind.

14 Name: #!/usr/bin/anonymous : 2006-12-05 14:12 ID:Heaven

>>13

No, feature bloat is not about useless features. It's about adding features that somebody wants, to the point where the program becomes less usable because there are so many special-purpose features. If it was just removing things nobody uses, it wouldn't be a problem.

15 Name: #!/usr/bin/anonymous : 2006-12-05 17:52 ID:/lRkt3Yv

>>12
My point is that modularization leading to bloat is more a matter of lacking discipline on the part of the programmer(s) or rather the designer(s), and not of modularization itself.

16 Name: #!/usr/bin/anonymous : 2006-12-05 19:55 ID:oUxukLrR

>>15
There you go. Agreed.

17 Name: #!/usr/bin/anonymous : 2006-12-06 02:15 ID:Heaven

>>15

With perfect discipline and insight, there'd never be a problem. That part is not really interesting. The point is, modularization and abstraction creates the temptation for feature bloat.

18 Name: #!/usr/bin/anonymous : 2006-12-06 03:41 ID:Heaven

>>9-17
Guns kill people.
No, people kill people.

19 Name: #!/usr/bin/anonymous : 2006-12-06 04:32 ID:Heaven

>>18

Don't start that "Oh no! ARGUMENT on the INTERNET!" shit. This isn't and argument that has been done to death, and it is entirely civil and well-reasoned.

Don't make a mockery of human discourse, man.

20 Name: #!/usr/bin/anonymous : 2006-12-06 08:12 ID:Heaven

>>19
That's just what Hitler would've said!

21 Name: #!/usr/bin/anonymous : 2006-12-06 10:33 ID:Heaven

I cant believe you people are actually discussing this. It makes no sense ! Do you seriously believe that it is better to make bad decisions, because good ones will allow you to add more features ?

Look, I'm not saying that every design should be perfect. If you really understand what you're doing, then there's nothing wrong with breaking the rules in the name of simpler/faster software ( which is the case here ).

22 Name: #!/usr/bin/anonymous : 2006-12-06 13:07 ID:Heaven

>>21

Maybe you are familiar with the gambit of playing Devil's Advocate?

But also, note how your own argument pre-supposes that modularization and abstraction is a good decision, and then argues that you should do this because it is a good decision. You are begging the question there.

This idea is very deeply ingrained in software developers. I am merely throwing out the suggestion here that maybe that isn't always the case.

23 Name: #!/usr/bin/anonymous : 2006-12-06 21:49 ID:oUxukLrR

I dunno. I tend to think if you lack discipline enough, you'll find a useless feature you can add instead of real work, regardless of how modular your code is or isn't. My own experience bloating up really messy FSMs attests to this.

24 Name: #!/usr/bin/anonymous : 2006-12-06 22:31 ID:Heaven

>>23

You code Flying Spaghetti Monsters? That's awesome.

Er, seriously though, you could argue that if it was hard enough to add those useless features, you would prefer to do the real work instead.

25 Name: #!/usr/bin/anonymous : 2006-12-07 01:55 ID:Heaven

[FSM means Finite State Machine.]

Wouldn't the real work be harder too?

26 Name: #!/usr/bin/anonymous : 2006-12-30 17:30 ID:Heaven

Delicious flat Guevara?

27 Name: #!/usr/bin/anonymous : 2007-01-02 08:31 ID:Heaven

Did someone mention Godwin's law yet?

28 Name: #!/usr/bin/anonymous : 2007-01-07 00:36 ID:Heaven

Saying modular design promotes feature bloat is like saying clean code and indentation promote feature bloat. The link is there if you squint hard enough, but it's sketchy at best.

Modular design may lead to a runtime performance hit, if it isn't automatically optimized out.

29 Name: #!/usr/bin/anonymous : 2007-01-12 02:33 ID:2YIGLePE

Database recoverability is much, much superior to filesystem recoverability. So at the very least, store your xml files in an RDB.

30 Name: #!/usr/bin/anonymous : 2007-01-12 23:15 ID:Heaven

>>29
And access the RDB with CORBA. Your methods should be wrapped at least five deep, or it won't be Enterprise and Scalable.

This thread has been closed. You cannot post in this thread any longer.