There's two ways you could algorithmically do what you want, both with different results.

The first is with a sinewave:

for (int i = 0; i < 360; i++) {
    analogWrite(5, (sin(i * 0.0.0174533) + 1) * 127));
    delay(3);
}

Basically you work your way 360 degrees around a circle and take one axis (e.g., the vertical one), offset it so that the values are all positive (sin gives a value between -1 and +1), then multiply it to fit the full brightness.

This is the smoothest way of fading an LED, but it doesn't respect the different sensitivities your eye has to the different brightnesses (you see more brightness variation when it's dim compared to when it's bright).

The other method is much cruder but does respect those differences in sensitivities. This specific method only gives you 8 brightnesses as opposed to 256, but they are on a log₂ curve:

for (int i = 0; i < 9; i++) {
    int v = (i << (i + 1)) - 1;
    analogWrite(5, v);
    delay(100);
}
for (int i = 7; i > 0; i--) {
    int v = (i << (i + 1)) - 1;
    analogWrite(5, v);
    delay(100);
}

The theory of this one is it uses bit-shifting to gradually fill a byte with 1 from the least significant bit upwards.

When i is 0, this formula:

int v = (i << (i + 1)) - 1;

gives:

0 + 1 == 1
1 << 1 == 0b00000001
0b00000001 - 1 = 0b00000000

If i is 4, you get:

4 + 1 = 5
1 << 5 == 0b00011111
0b00011111 - 1 = 0b00001111

So you effectively end up with a set of values:

0: 0
1: 1
2: 3
3: 7
4: 15
5: 31
6: 63
7: 127
8: 255

There's two ways you could algorithmically do what you want, both with different results.

The first is with a sinewave:

for (int i = 0; i < 360; i++) {
    analogWrite(5, (sin(i * 0.0174533) + 1) * 127));
    delay(3);
}

Basically you work your way 360 degrees around a circle and take one axis (e.g., the vertical one), offset it so that the values are all positive (sin gives a value between -1 and +1), then multiply it to fit the full brightness.

This is the smoothest way of fading an LED, but it doesn't respect the different sensitivities your eye has to the different brightnesses (you see more brightness variation when it's dim compared to when it's bright).

The other method is much cruder but does respect those differences in sensitivities. This specific method only gives you 8 brightnesses as opposed to 256, but they are on a log₂ curve:

for (int i = 0; i < 9; i++) {
    int v = (1 << i) - 1;
    analogWrite(5, v);
    delay(100);
}
for (int i = 7; i > 0; i--) {
    int v = (1 << i) - 1;
    analogWrite(5, v);
    delay(100);
}

The theory of this one is it uses bit-shifting to gradually fill a byte with 1 from the least significant bit upwards.

When i is 0, this formula:

int v = (i << i) - 1;

gives:

1 << 0 == 0b00000001
0b00000001 - 1 = 0b00000000

If i is 4, you get:

1 << 4 == 0b00010000
0b00010000 - 1 = 0b00001111

So you effectively end up with a set of values:

0: 0
1: 1
2: 3
3: 7
4: 15
5: 31
6: 63
7: 127
8: 255