How to?

1. Install Nginx
2. Configure Nginx (see configuration file below)
3. Make sure your backend server runs otherwise you’ll be welcomed by an error message 512 bad gateway, e.g. start fastcgi with manage.py

Nginx configuration file

Tested on Ubuntu Linux 8.04

worker_processes 1;
error_log logs/error.log info;
pid /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include mime.types;
    default_type application/octet-stream;

    server {
        listen 80;
        server_name localhost;
        location / {
            # Launch proxied server on a separate port
            proxy_pass http://127.0.0.1:8000;
            proxy_set_header   Host             $host;
            proxy_set_header   X-Real-IP        $remote_addr;
            proxy_set_header   X-Forwarded-Host $server_name;
            proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        }
        location /media/  {
            # Symlink to django admin media files
            alias /source/media/;
        }
        location /static/ {
            # static files for the project
            root  /clients/goto;
        }
    }
}

To add a new site you need to add another Nginx server{} to your config file and run another instance of the backend server.

You can use reverse proxy with Apache/mod_python, FastCGI or anything else. I’ve used this with the WSGI server extracted from the CherryPy and it works like charm.

This is also my preffered Django development setup. The development server is much faster if it doesn’t have to serve static files. Just proxy Nginx to the development server.

The problem happened as I separated static media files to another server. Separating static files such as Javascript, CSS and images is a recommended step when scaling a website on a threaded server without proxy load balancers.

This WYSYWIG and all similar editors suck for the following reasons:

• It is slow to load as it needs a bazillion of files
• It needs to be on the same server (same domain and same domain level to be exact) as the generated HTML, which brakes it with advanced server setups that need static files on a separate server instance (You can partly fix this with advanced server side URL redirections)
• last but not least: It lets any Joe and Jane paste messy HTML tags from Word, which brakes the type consistency and generally brakes beauty of the hypertext in various unimaginable ways

However, TinyMCE is one of the best options on this niche market as it allows any Joe and Jane to shoot themselves in the foot with HTML non-data tables. Unfortunately every client requires to be able to do this.

WYSIWYG should be a browser independent extension users can use for any text-area that allows HTML input.

There is still hope. Until someone will open-source solution such as above, developers can use WYM.

There is also a fix for all the fine developers who stumbled across a similar TinyMCE error. I guess there is at least 200,000 of you:

Editor’s note: » marks added line break

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteRule ^path-to-tinymce/(.*)$ »
http://staticfile.net/path-to-tinymce/$1 [R=301,NC,L]

Disclosure: I am not dissing just TinyMCE, I am dissing every WYSYWIG that works on a similar principle.

Frank Malina

Article graciously provided by Sylvia Hausnerova, edited by Frank Malina.

Data mining is a process of analysis of a data set aiming

1. to answer a predefined question
2. to recognize a pattern within a data set

Data-mining process

Application should keep in mind the following user-flow:

1. Set a question you need to answer.
2. Issue queries on a dataset that will give you insight on your question.
3. Take actions to take advantage of the new insight.

OR a more casual work flow:

1. Browse the dataset in various ways.
2. Try to recognize patterns.
3. Take actions to take advantage of the recognized patterns.

What can data mining do?

Data mining is commonly used in analysis of web site traffic data. Traffic data analysis can give you useful marketing and sales related data.

Such as:

• When do people shop in my e-commerce store?
  When should I bid on ads displayed in Google to increase traffic to my site for cheaper?
• Where from are my customers?
  Where should I invest to off-line marketing to make my brand stronger?
• What browsers my users use?
  Should I invest money in testing the design of my site in particular web browser? Will I loose money for not testing my design in IE5.

Most commonly used tool for analysis of web traffic data as mentioned above is Google analytics.

However what it takes to create a CEO-friendly tool allowing to analyze custom data sets such as off-line sales data?

What technology to create Analytics for custom data sets?

Generally there a tree layers of software to be written

1. Data warehouse - a database that stores the collected data (MySQL, PostgreSQL, Oracle to name a few). This is probably in place already.
2. Interface for data queries - SQL object is a great tool allowing to create interfaces for data mining. Most of the functionality of SQL object is now baked in the Django web application framework.
3. Interfaces that display the data retrieved from the warehouse in a human readable form (pie charts, tables, graphs). There a various techniques to make this happen such as open flash chart, CSS bar graphs and HTML tables enhanced with AJAX scripting.

I wrote down my notes on a solution that will help to keep the American network freedom.

It’s ridiculous that American citizens have even to worry about this issue.

Just to worry Americans less, I wrote down my notes on a technical solution that will help to keep the current status quo of the American network freedom.

I’d imagine that this will be something integrated as an Firefox/Opera/IE7 add-on that seamlessly integrates with the browser. User won’t even recognize that the add-on is in use.

Behind the scenes:

• Browser add-on will get a proxy server assigned based on the load-balancing algorithm with each request
• Add-on will hook up into the navigation bar, ensuring that each request is send to the user’s original ISP first
• If the content is blocked (404, or redirect detection), or there is a speed limitation (network speedtest), add-on will request the proxy to check that the content is truly there
• If the content exists the Proxy server will request the content on the user’s behalf and feed it to the client
• Bandwidth of proxies and the infrastructure maintenance will be quite expensive
• We could save money on the infrastructure maintenance by open-sourcing it
• The bandwidth costs could be partially covered by blocking the bandwidth expensive graphical banner ads on the sites, replacing it with fast-loading relevant text links. A lot of work has been done on the ad detection already, so it looks doable. Content providers might not like it though.

Proxies are used in China, where CISCO blocked a reasonable part of the network. I’ve read it works, unless you’ll get caught.

Anyway, shouldn’t this two-tiered service be impossible just form the market-driven economy point of view? Would you buy a castrated internet connection? No way. Those ISPs that won’t provide full access are out of business via competition that gives its users the full online freedom.

If the American market is not competitive enough… we’ll have some fun with the above.

Never changing best practices

• Good Microcontent
• Scannable layout
• Concise / objective language

Microcontent: page title tags, headlines

Title tags, headings should make sense out of context.

Microcontent needs to be pearls of clarity. You get up to 65 characters to explain your macrocontent.

Titles longer than 65 characters are concatenated in search results, so don’t waste your ink.

Unless the title or heading make it absolutely clear what the page is about, users will never read it, because they won’t find it in Google.

Headings are often displayed out of context. Headline text has to stand on its own. Written in plain language. Make the first word an important, information-carrying one.

Scannable text structure

• Use rich semantic markup
• Meaningful sub-headings (not “clever” ones)
• use unordered and ordered lists, where appropriate
• one idea per paragraph (users will skip over any additional ideas if they are not caught by the first few words in the paragraph)
• the inverted pyramid style, starting with the conclusion
• Highlighted keywords: hypertext links serve as one form of highlighting; use bold for <strong> emphasis and italic for disturbing-kind-of-<em>phasis.

Concise / objective language

Users appreciate professional advice, not promotional language. Keep commercial language to a minimum. Credibility suffers when readers (the human scanners) clearly see your writing exaggerates. Promotional language has naturally far lower density of common search phrases. If you’re in a hurry and can hear bla, bla ,bla.. when you read the copy, it means that the content is very poor on search phrases.

Useful information

• credible facts
• tips and tricks
• bits of know-how

A wall of text is deadly for an interactive experience. Search engines employ clever algorithms (and a lot of human beings) to find out whether your content is useful or a spam. Keep in mind the language of searchers when writing for the web.

Inverted Pyramid writing style

Start with conclusions or short summary and continue with generic info. further down the copy. Users (and Search engines) consider top of the article more important or at least they read it first. First and last paragraphs need to hold the semantic value, search phrases. This is completely different compared to printed media, where conclusions are usually on the end of the article.

Edit: PayPal have fixed the issue. Well done PayPal! Their site now checks for the PayPal login cookie. If you don’t have one, it presumes you are not a regular PayPal user and returns a complete form. You can still use the below to learn how to make an elegant PayPal donation link without involving messy HTML form markup.

The link syntax is very simple: https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=***INSERT YOUR EMAIL HERE***&item_name=***YOUR PROJECT/CAMPAIGN/CHARITY... NAME HERE*** - HINT: TRY TO FIND ON THE SCREEN A TINY LINK TO >CONTINUE< CREDIT CARD PAYMENT&amount=***INSERT INTEGER FOR THE AMOUNT HERE***&currency_code=***CURRENCY HERE***

Note the added hint for the user to click continue link to display the credit card form. The text is a bit cheeky but it illustrates the issue well. Edit: Forget about that bit now.

Every commercial merchant or non-profit organization can use my simple technique to increase conversions. This technique doesn’t break PayPal terms of service (I’ve read carefully). It doesn’t break PayPal undefeated security. It’s just a tiny clever trick in fact.

]]> http://vizualbod.com/articles/paypalhack/feed http://vizualbod.com/articles/makeyourownyoutube http://vizualbod.com/articles/makeyourownyoutube#comments Thu, 03 Jul 2008 12:16:21 +0000 Frank Malina http://vizualbod.com/?p=48 The video streaming application setup based on my assumptions to explain what a developer needs to consider from the technical/scalability point of view  to create video streaming social media website such as YouTube, Break.com or Google video.

Video precessing

Conversion of various uploaded video file types to the FLV (Flash video streaming format)

Mature video conversion classes are available in public license for Python programming language. Pyhton or C will make the processing more real-time although a processing queue is very likely to happen. Resulting FLVs are stored on a permanent storage device.
Video precessing server certainly needs a massive processing power.

Application server

Django/RoR or other web frameworks will be useful at this point. I would prefer to use Django framework. There is a built in object caching and support for memcached. Heavy load is to be expected as a nature of the video streaming sites and caching at various levels of the process is very handy.
This needs to be engineered to expect load on the certain videos, while other videos will have hardly any views. Caching will make sure that the pages for videos under heavy load are served instantly, freeing up the resources on the machine.

Database server

Running MySQL or PosgreSQL instance serving user posts and and comments controllers’ textual data requested from the website processing server.

Static content server

Serving thumbnails, cached pages, streaming video files

Nature of video files is that they are big, therefore this machine(s) will need a some disk space, separate from the three machines above. Nginx is a good hint for serving video thumbnails. This server will also serve the JavaScript, CSS files, Flash action scripts + design related imagery. This should also account for the scheduled backups to a tape storage device as the project grows (not necessarily at the start).

See Nginx proxy to the application server for a good hint on serving static content.

Database server, website precessing & video precessing and static content parts will need to be loosely coupled allowing the developer to separate them and replicate setups on more physical machines as the site grows.