algorithm analysis - Mon Feb 22

Questions about anything?

Discuss the homework.

I've posted an assignment for next Monday. We'll see how far we get through these topics today, and continue on Thursday.

analysis of algorithms & O()

This week the goal is to understand how we measure an algorithm's behavior. What makes one "good"? What does that even mean - easy? fast? small?

The basic ideas are

• time (how long does it take) & space (how much menory)
• These depend on the "model of computation" and computer hardware. Which operations are expensive in time & space?
• The problems we're interested in all have a "size" N
• These days, for most of the problems we see, time is a bigger constraint than space.
• ... and so what we're usually most interested in is how fast the algorithm runs as N gets big.

Often times there's a tradeoff between time and space ... we can find a faster method if we use more memory.

What sort of algorithms are we talking about?

Example : shortest robot traversal (pg 16+)

Explain O() with Skienna's chap 2 slides (pg 5+)

In class exercise ... "Write two functions to find the minimum number in a list ... O(n**2), O(n)" ... here's my version

So how do we learn what what O() is for a given algorithm?

• (a) thinking about loops and and how many times through them
• (b) numerical experiments ... counting lines executed OR timing explicitly

For a given problem? Answer: it's hard to know ... maybe we just haven't found the right algorithm.

Here's an example of several O() algorithms for the same problem : is_anagram(word1, word2)

Another example of O(log n) : binary search.

Which algorithm is "best" may depend also on how many times operation will be repeated, or memory requirements, or how much preparation is needed. If for example you want to search for one person given a million of them, you might (a) do an O(n) linear search if you are only going to search once in an unordered collection, or (b) sort them (which takes O(n log(n)) and then do repeated binary searches O(m log n) for m repetitions, or (c) put them all into a dictionary to get O(1) search ... but at the cost of lots of memory and O(n) filling in dictionary.

"easy" vs "hard"

Turns out that in practice, there's a big difference between exponential algorithms (2^N, N!, ...) and polynomial (n**2, n log(n), ...) algorithms. I'm oversimplifying, but the exponential ones are in general too slow for any sizes that aren't tiny. Problems whose known algorithms are exponential are considered "hard".

mathy stuff

Review log functions : log2(n), log10(x) ... see for example khan academiy

• Why : because many of the important algorithms manage to reduce N steps to log(N) steps instead
• Often by tricks like "divide-and-conquer" : example - binary search.

Visualizing numerical data with plots (jupyter python notebook, numpy, matplotlib) - an example

Counting :

• all pairs of N things
• all sequences of N things
• all True/False assignments to N variables

where we're headed

Understand O() behavior for various operations for data structures containers : arrays, lists, dictionaries, etc. Typically there are tradeoffs; you cannot optimize for all types of operations such as (add data, remove data, search for data, get smallest data, merge data, etc).

Use those data structures as components in recipes for specific problems : searching, graphs, ...

Find common ideas in different algorithms: brute-force, divide-and-conquer, greedy, dynamic-programming, ...

Next week : a few common data structures, revisited.

Coming : sorting ... lots of different ways.